You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "pratham76 (via GitHub)" <gi...@apache.org> on 2026/09/09 09:10:26 UTC
[PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
pratham76 opened a new pull request, #58656:
URL: https://github.com/apache/spark/pull/58656
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'common/utils/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
`RewritePredicateSubquery` turns a predicate subquery in a `Filter` into a semi/anti join, using
the correlated predicates that `PullupCorrelatedPredicates` hoisted out of the subquery as the
join condition. Nested existential subqueries in that condition were all rewritten into existence
joins on top of the **outer** plan. This PR rewrites each of them against the plan that actually
produces the attributes it references:
* references the subquery plan and not the outer plan -> the existence join is built on top of the
subquery plan;
* otherwise -> on top of the outer plan, as before;
* references both plans -> left in the join condition, where both plans are in scope (it cannot be
rewritten on either side).
Concretely, `rewriteExistentialExprWithAttrs` gains a `canRewrite` predicate so a caller can
rewrite a subset of the nested subqueries, and two helpers
(`rewriteExistentialExprInJoinCondition` / `rewriteExistentialExprInSubqueryPlan`) route each
nested subquery to the correct side. All four semi/anti join rewrites (`EXISTS`, `NOT EXISTS`,
`IN`, `NOT IN`) use them.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
A correlated predicate that is a disjunction is hoisted out of the subquery as a whole, so it can
carry a nested existential subquery with it. That nested subquery may reference the subquery plan,
in which case building its existence join on top of the outer plan produces a join whose condition
references an attribute that neither of its children can produce, i.e. a plan node that references
non-reachable attributes:
```sql
select *
from (values (1), (2), (3)) as t1(a)
where exists (
select c1
from (values (1), (8), (9)) as t2(c1)
where a = c1
or c1 in (select col1 from (values (3), (9)) as t3(col1))
);
```
```
[INTERNAL_ERROR] Couldn't find c1#25 in [a#24,col1#26] SQLSTATE: XX000
```
The optimized plan shows the invalid node (`!Join`), where the existence join for
`c1 IN (SELECT col1 FROM t3)` sits on top of the outer plan and its condition references `c1`,
which is only produced by the subquery plan on the right of the semi join:
```
Project [a#9]
+- Join LeftSemi, ((a#9 = c1#10) OR exists#12)
:- !Join ExistenceJoin(exists#12), (c1#10 = col1#11)
: :- LocalRelation [a#9]
: +- LocalRelation [col1#11]
+- LocalRelation [c1#10]
```
With this PR the existence join is built on top of the subquery plan, which produces `c1`:
```
Project [a#9]
+- Join LeftSemi, ((a#9 = c1#10) OR exists#12)
:- LocalRelation [a#9]
+- Join ExistenceJoin(exists#12), (c1#10 = col1#11)
:- LocalRelation [c1#10]
+- LocalRelation [col1#11]
```
and the query returns `1, 2, 3`, which is what PostgreSQL and DuckDB return.
The same failure affects `NOT EXISTS`, `IN` and `NOT IN` subqueries, and nested `IN`, `NOT IN` and
`EXISTS` subqueries; a nested subquery correlated to the subquery it is nested in produced the
same invalid plan but happened to avoid the error when the nested relation was empty at runtime,
because the invalid condition was then never bound.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes. Queries of the shape above failed with `INTERNAL_ERROR` and now return
the correct result.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Existing UTs, additional tests have been added
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Claude Code (Claude Sonnet 4.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4040725362
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -439,7 +454,8 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), Some(finalJoinCond), joinHint)
introducedAttrs += exists
Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
Review Comment:
Not intended — thanks for spotting it. `transformDownWithPruning` recurses into the children even
when the rule returns the node unchanged (it compares with `fastEquals` and still calls
`mapChildren`), so returning the node as-is was not enough to stop the descent. Since
`SubqueryExpression.children` is `outerAttrs ++ joinCond`, a sub-query nested in a declined
leftover's own join condition was rewritten, grafting an existence join for an `exists` reference
that lands in a join condition which `PlanSubqueries` may drop — I could see it in the plan for your
doubly nested shape:
```
list#128 [c1#131 && c1#131 && ((col1#132 = c1#131) OR exists#140)]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4148344182
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
Review Comment:
Fixed in the scaladoc and in the test comment: cannot be rewritten on either side, *therefore* it
has to stay in the join condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6391596365
Thanks, I checked 289efbb46d9. An uncorrelated carried IN with an observable unknown is now rejected with NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT under both settings, the two carried rejections raise their own conditions, the new tests cover the Filter-predicate, projection and join-condition routes, and the NOT IN residual cases now discriminate. A carried NOT IN takes the null-aware arm on the reported shape, but not when the same IN sub-query is also carried bare; that is in a new comment on rewriteCarriedExistentialExprs in optimizer/subquery.scala. Also still open from that round: the NOT IN arm's comment there still says keeping the residual is correct and treats a wrapped anti join as hypothetical, and only the test comment was reworded.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"conversation:6380316971","thread_id":"conversation:6380316971","verdict_sha256":"13b761b833eea68b81b1a432a43d105b57e032a93d09a2fe8b2af5e80ef3d168"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4248986922
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,412 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `attr` is produced by `outerPlan` and not by `subPlan`.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`.
+ * This matters when the two share ExprIds, which a self-join can leave for the optimizer to
+ * deduplicate (SPARK-21835): the EXISTS arms classify against the raw sub-query plan, while the
+ * IN arms classify against the plan that `dedupSubqueryOnSelfJoin` has already separated from
+ * the outer plan. Without this a single shared attribute would satisfy both sides of the
+ * classification, so a nested sub-query that references only the sub-query it is nested in
+ * would be taken for one referencing both plans and rejected. A nested sub-query resolves in
+ * the scope of that sub-query, so a shared attribute is one of its own.
+ */
+ private def referencedByOuterPlanOnly(
+ attr: Attribute,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): Boolean = {
+ outerPlan.outputSet.contains(attr) && !subPlan.outputSet.contains(attr)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
Review Comment:
**Nit (P3):** This rationale contradicts the implementation: collectUnknownSensitiveInSubqueries begins with a containsAnyPattern guard and therefore prunes every subquery-free subtree. Describing both collectors as unpruned gives future changes the wrong cost model for this path.
**Verification:**
- **Inspection:** Compare each traversal claim with the top-level containsAnyPattern guard and the recursive collector's subtree guard.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,412 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
Review Comment:
**Blocking (P1):** Returning a declined parent here without inspecting its children can hide a subplan-owned correlated IN below an uncorrelated parent that references both plans. The rejection pass accepts that child because its own effective references are inner-only, and PlanSubqueries later ignores ListQuery.joinCond, so the correlation predicate can be lost and the query can silently return extra rows instead of raising the structured unsupported-query error.
**Recommended change:** Make residual validation aware of rewrite stopping boundaries. Before leaving a declined existential parent intact, identify correlated descendants in every effective carrier that will remain blocked and reject any whose join condition downstream planning cannot preserve. Keep ordinary subplan-owned descendants rewriteable when they are actually reached, and keep the accepted uncorrelated both-plan residual behavior.
**Why this works:** Use one carrier-aware traversal state that records whether a descendant is reachable by the custom rewrite or blocked beneath a declined parent. Apply ownership routing to reachable nodes and the structured correlation rejection to blocked correlated nodes, so no correlated ListQuery can survive to PlanSubqueries with semantic state stored only in joinCond.
**Scope:** Align residual-subquery validation with the custom rewrite's no-descent boundary and cover the hidden correlated composition across configuration modes.
**Compatibility:** The custom rewrite still avoids grafting existence joins into declined subqueries, and supported direct nested-subquery routing remains attribute-owner based.
**Risks:** An overbroad scan can reject a descendant that the rewrite would safely consume. A generic descent can reintroduce existence joins whose replacement attributes live only in discarded subquery state.
**Constraints:** Uncorrelated both-plan subqueries retain their accepted residual behavior under the documented configuration values. Simplified-away correlation and retained outerAttrs continue to use effective rather than stale references. Direct nullable IN/NOT IN truth-only and value-sensitive handling remains unchanged.
**Success:** No correlated descendant blocked beneath a declined existential parent reaches physical planning with its semantics stored only in joinCond. The distinguishing non-null example returns only a=1 or receives the intended structured unsupported-query error, never all rows. Ordinary reachable subplan-owned and outer-owned nested subqueries continue to route to the plan supplying their effective references.
**Verification:**
- **Regression:** Exercise a both-plan uncorrelated parent IN carrying a subplan-owned correlated child with distinguishing non-null data and verify correlation is retained or structurally rejected under disabled join-predicate decorrelation.
- **Compatibility:** Verify the same composition under relevant configuration values while preserving accepted uncorrelated both-plan, simplified-correlation, direct routing, and nullable-IN behavior.
- **Inspection:** Verify every residual path to physical subquery planning has either empty correlation state or an optimizer representation that preserves it.
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,375 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: nested IN subquery whose result can be unknown") {
+ withTempView("t1", "t2", "t2n", "t3n", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // A nullable compared value, against a subquery column that is not nullable.
+ Seq(Some(1), None).toDF("c1").persist().createOrReplaceTempView("t2n")
+ Seq((1, 1), (9, 9)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ // The NULL matches no value of t2, so c1 IN (SELECT col1 FROM t3n) is unknown there, and
+ // c1 NOT IN (SELECT col1 FROM t3n) is never true.
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // An existence join only records whether a row matched, so its exists attribute is false
+ // where IN is unknown. That is not observable while the value feeds a predicate, but it is
+ // once it reaches anything else, so these are rejected rather than rewritten. Both
+ // configurations are checked, as the rejection must not depend on the rewrite of predicate
+ // subqueries in join conditions.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> false",
+ "(c1 IN (SELECT col1 FROM t3n)) IS NULL",
+ // COALESCE to a non-false default does tell unknown from false.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), true)",
+ "(c1 NOT IN (SELECT col1 FROM t3n)) <=> false").foreach { predicate =>
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)")
+ .collect()
+ }
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT")
+ // A NOT IN must be reported as a NOT IN, not as the IN underneath it.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.startsWith("(NOT ") == predicate.contains("NOT IN"),
+ s"unexpected subqueryExpression for $predicate: $reported")
+ }
+ }
+ }
+
+ // The ordinary case: the value of the nested IN feeds a predicate, where unknown and
+ // false cannot be told apart, so it is still rewritten even though it can be unknown.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t3n))""".stripMargin,
+ Row(1) :: Nil)
+
+ // And a nested IN that cannot be unknown is rewritten even outside a predicate.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3)) <=> false))""".stripMargin,
+ Row(1) :: Row(2) :: Row(3) :: Nil)
+
+ // Only truth is observed by `<=> true`, by an IF predicate and by a CASE WHEN
+ // condition, so those are rewritten rather than rejected. SimplifyConditionals folds
+ // the latter two into the first, so ordinary SQL reaches this position.
+ //
+ // The result branches of a conditional are consumed wherever the conditional is, so
+ // under a filter predicate they are truth-only as well. The last two cases do not
+ // fold, so they exercise that inheritance rather than the rewrites above.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> true",
+ "IF(c1 IN (SELECT col1 FROM t3n), true, false)",
+ "CASE WHEN c1 IN (SELECT col1 FROM t3n) THEN true ELSE false END",
+ "IF(c1 > 0, c1 IN (SELECT col1 FROM t3n), false)",
+ "CASE WHEN c1 > 0 THEN c1 IN (SELECT col1 FROM t3n) ELSE false END",
+ "CASE WHEN c1 > 100 THEN false ELSE c1 IN (SELECT col1 FROM t3n) END",
+ // COALESCE to false maps unknown and false alike, so only truth is observed.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), false)")
+ .foreach { predicate =>
+ checkAnswerAndPlan(
+ s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)",
+ Row(1) :: Nil)
+ }
+
+ // The unknown can come from the compared value rather than from the subquery column.
+ // spark.sql.legacy.inSubqueryNullability makes InSubquery.nullable ignore the
+ // nullability of the list query columns, but it always keeps that of the compared
+ // values, and the guard re-derives both, so this case is rejected under either value.
+ Seq("true", "false").foreach { legacyNullability =>
+ withSQLConf(SQLConf.LEGACY_IN_SUBQUERY_NULLABILITY.key -> legacyNullability) {
+ val nullableValue = intercept[AnalysisException] {
+ sql("""SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2n
+ | WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3)) <=> false))""".stripMargin)
+ .collect()
+ }
+ assert(nullableValue.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT")
+ }
+ }
+
+ // An IN subquery nested in the values or in the hoisted condition of another one is
Review Comment:
**Non-blocking (P2):** This comment overstates the regression coverage: nestedTwoLevels reaches the inner IN through the outer ListQuery.joinCond, so it never exercises the separate in.values descent. Removing that branch leaves this test green even though an unknown-producing IN used as another IN's compared value can distinguish UNKNOWN from FALSE and expose a wrong result.
**Verification:**
- **Regression:** Verify a nested IN in compared values distinguishes UNKNOWN from FALSE and fails if values-side recursion is removed.
- **Compatibility:** Verify the existing nested join-condition rejection and direct NOT IN behavior remain unchanged.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,412 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `attr` is produced by `outerPlan` and not by `subPlan`.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`.
+ * This matters when the two share ExprIds, which a self-join can leave for the optimizer to
+ * deduplicate (SPARK-21835): the EXISTS arms classify against the raw sub-query plan, while the
+ * IN arms classify against the plan that `dedupSubqueryOnSelfJoin` has already separated from
+ * the outer plan. Without this a single shared attribute would satisfy both sides of the
+ * classification, so a nested sub-query that references only the sub-query it is nested in
+ * would be taken for one referencing both plans and rejected. A nested sub-query resolves in
+ * the scope of that sub-query, so a shared attribute is one of its own.
+ */
+ private def referencedByOuterPlanOnly(
+ attr: Attribute,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): Boolean = {
+ outerPlan.outputSet.contains(attr) && !subPlan.outputSet.contains(attr)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
Review Comment:
**Nit (P3):** This effective-reference set is rebuilt and rescanned across both-plan rejection, UNKNOWN filtering, and rewrite ownership checks. For a wide row-valued IN, each call first flattens child AttributeSets into a cardinality-sized sequence and then performs separate membership scans, adding repeated optimizer allocation and CPU for the same candidate.
**Verification:**
- **Inspection:** Trace a wide nested expression through all classification phases and verify no cardinality-sized flattened sequence or repeated reference-set reconstruction remains.
- **Compatibility:** Verify ownership and rejection results for inner-only, outer-only, both-plan, unreferenced, shared-ExprId, and simplified-away-correlation shapes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6189163137
Thanks for the update. I re-reviewed 575528ff4b7, and the r4139416473 request is in: SizeInBytesOnlyStatsPlanVisitor and JoinEstimation now both estimate an ExistenceJoin from its left child, and the new SubquerySuite test pins that the enclosing semi join keeps its broadcast.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"conversation:6183220982","thread_id":"conversation:6183220982","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4286966748
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,540 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: nested IN subquery whose result can be unknown") {
+ withTempView("t1", "t2", "t2n", "t3n", "t3", "t4", "tb", "t2r", "t2rn", "t6", "t7") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // A nullable compared value, against a subquery column that is not nullable.
+ Seq(Some(1), None).toDF("c1").persist().createOrReplaceTempView("t2n")
+ Seq((1, 1), (9, 9)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ // A boolean relation, to compare the result of a nested IN subquery against.
+ Seq(false).toDF("b").persist().createOrReplaceTempView("tb")
+ // Row-valued shapes. t2r and t7 are not nullable anywhere; the nullability of t2rn and of
+ // t6 sits in the second field, so the first pair of the comparison does not expose it.
+ Seq((1, 1), (8, 8), (9, 9)).toDF("c1", "c2").persist().createOrReplaceTempView("t2r")
+ Seq((1, Option.empty[Int]), (8, Some(8))).toDF("c1", "c2n").persist()
+ .createOrReplaceTempView("t2rn")
+ Seq((1, Option.empty[Int]), (5, Some(5))).toDF("col1", "col2").persist()
+ .createOrReplaceTempView("t6")
+ Seq((1, 1), (5, 5)).toDF("col1", "col2").persist().createOrReplaceTempView("t7")
+ // The NULL matches no value of t2, so c1 IN (SELECT col1 FROM t3n) is unknown there, and
+ // c1 NOT IN (SELECT col1 FROM t3n) is never true.
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // An existence join only records whether a row matched, so its exists attribute is false
+ // where IN is unknown. That is not observable while the value feeds a predicate, but it is
+ // once it reaches anything else, so these are rejected rather than rewritten. Both
+ // configurations are checked, as the rejection must not depend on the rewrite of predicate
+ // subqueries in join conditions.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> false",
+ "(c1 IN (SELECT col1 FROM t3n)) IS NULL",
+ // COALESCE to a non-false default does tell unknown from false.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), true)",
+ "(c1 NOT IN (SELECT col1 FROM t3n)) <=> false").foreach { predicate =>
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)")
+ .collect()
+ }
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT")
+ // A NOT IN must be reported as a NOT IN, not as the IN underneath it.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.startsWith("(NOT ") == predicate.contains("NOT IN"),
+ s"unexpected subqueryExpression for $predicate: $reported")
+ }
+ }
+ }
+
+ // The ordinary case: the value of the nested IN feeds a predicate, where unknown and
+ // false cannot be told apart, so it is still rewritten even though it can be unknown.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t3n))""".stripMargin,
+ Row(1) :: Nil)
+
+ // And a nested IN that cannot be unknown is rewritten even outside a predicate.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3)) <=> false))""".stripMargin,
+ Row(1) :: Row(2) :: Row(3) :: Nil)
+
+ // Only truth is observed by `<=> true`, by an IF predicate and by a CASE WHEN
+ // condition, so those are rewritten rather than rejected. SimplifyConditionals folds
+ // the latter two into the first, so ordinary SQL reaches this position.
+ //
+ // The result branches of a conditional are consumed wherever the conditional is, so
+ // under a filter predicate they are truth-only as well. The last two cases do not
Review Comment:
Correct -- "the last two" pointed at the ELSE-branch CASE and `COALESCE(..., false)`, and the latter is not a conditional at all and has its own comment. All three result-branch cases stay unfolded, since `SimplifyConditionals` folds an `If` or `CaseWhen` only when its branches are TRUE/FALSE literals, or null literals with a non-nullable condition. The comment now names the three result-branch cases and gives that folding rule as the reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4130355286
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,372 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, the predicate of an `If` and the conditions of a `CaseWhen`.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) => Nil
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
+ unknownSensitiveInSubqueries(right, truthOnly)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
+ unknownSensitiveInSubqueries(right, truthOnly)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here. The
+ // same holds for the branch conditions of a conditional.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ unknownSensitiveInSubqueries(left, truthOnly = true)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ unknownSensitiveInSubqueries(right, truthOnly = true)
+ case If(predicate, trueValue, falseValue) =>
+ unknownSensitiveInSubqueries(predicate, truthOnly = true) ++
+ unknownSensitiveInSubqueries(trueValue, truthOnly = false) ++
Review Comment:
**Non-blocking (P2):** When the enclosing `If` is itself consumed in a truth-only position, its result branches inherit that context: NULL and FALSE from either branch are indistinguishable to the filter. Resetting `truthOnly` here causes a non-foldable expression such as `IF(c1 > 0, c1 IN (...), false)` to be rejected even though lowering that IN to an existence bit is exact. Please propagate the incoming context to the true and false values, and do the same for `CaseWhen` result and else branches, with non-foldable coverage for those paths.
**Verification:**
- **Regression:** Exercise non-foldable If true/false values and CaseWhen result/else values containing nullable inner-routed IN expressions under an enclosing filter predicate, and assert both answers and subquery-side plan placement.
- **Compatibility:** Retain rejection of nullable IN under value-sensitive parents such as <=> false and preserve the established direct NOT IN result behavior.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,372 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, the predicate of an `If` and the conditions of a `CaseWhen`.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) => Nil
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
+ unknownSensitiveInSubqueries(right, truthOnly)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
+ unknownSensitiveInSubqueries(right, truthOnly)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here. The
+ // same holds for the branch conditions of a conditional.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ unknownSensitiveInSubqueries(left, truthOnly = true)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ unknownSensitiveInSubqueries(right, truthOnly = true)
+ case If(predicate, trueValue, falseValue) =>
+ unknownSensitiveInSubqueries(predicate, truthOnly = true) ++
+ unknownSensitiveInSubqueries(trueValue, truthOnly = false) ++
+ unknownSensitiveInSubqueries(falseValue, truthOnly = false)
+ case CaseWhen(branches, elseValue) =>
+ branches.flatMap { case (condition, value) =>
+ unknownSensitiveInSubqueries(condition, truthOnly = true) ++
+ unknownSensitiveInSubqueries(value, truthOnly = false)
+ } ++ elseValue.toSeq.flatMap(unknownSensitiveInSubqueries(_, truthOnly = false))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) =>
+ collectOrDescend(not, in, truthOnly)
+ case Not(child) => unknownSensitiveInSubqueries(child, truthOnly = false)
+ case in: InSubquery =>
+ collectOrDescend(in, in, truthOnly)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.flatMap(unknownSensitiveInSubqueries(_, truthOnly = true)) ++
+ sq.getOuterAttrs.flatMap(unknownSensitiveInSubqueries(_, truthOnly = false))
+ case other =>
+ other.children.flatMap(unknownSensitiveInSubqueries(_, truthOnly = false))
+ }
+
+ /**
+ * Collects `reported` if the IN sub-query `in` would lose an observable unknown, and otherwise
+ * descends into the values it compares and into its list query. The descent matters because
+ * [[InSubquery]] is a [[Predicate]] rather than a [[SubqueryExpression]], so an IN sub-query
+ * nested in the values or in the hoisted join condition of another one is only reached from
+ * here.
+ */
+ private def collectOrDescend(
+ reported: Expression,
+ in: InSubquery,
+ truthOnly: Boolean): Seq[Expression] = {
+ if (!truthOnly && inSubqueryMayBeUnknown(in)) {
+ Seq(reported)
+ } else {
+ in.values.flatMap(unknownSensitiveInSubqueries(_, truthOnly = false)) ++
+ unknownSensitiveInSubqueries(in.query, truthOnly)
+ }
+ }
+
+ /**
+ * Returns true if the row comparison of the IN sub-query `in` can evaluate to unknown, that is
+ * if it can return NULL rather than only TRUE or FALSE. An existence join cannot represent that
+ * third value, see [[rewriteExistentialExprInJoinCondition]].
+ *
+ * This deliberately does not use `InSubquery.nullable`, which returns false for either side
Review Comment:
**Nit (P3):** Legacy mode does not make `InSubquery.nullable` false for either side. Its legacy branch is `values.exists(_.nullable)`, so nullable compared values remain nullable; only list-query output nullability is ignored. Please update this explanation and the corresponding test comment, fix the `configuration own trade-off` typo, and cover the nullable-left case under both legacy settings so this boundary is explicit.
**Verification:**
- **Compatibility:** Verify that a nullable compared value with non-nullable list output reaches the UNKNOWN guard under both legacy settings and receives the same structured rejection.
- **Inspection:** Confirm every new legacy-nullability comment matches the implemented contract: values nullability is always considered and list-query output nullability is omitted only in legacy mode.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,372 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, the predicate of an `If` and the conditions of a `CaseWhen`.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) => Nil
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
Review Comment:
**Nit (P3):** Because these results are immutable sequences, combining recursively returned prefixes with `++` at every `And` or `Or` node repeatedly copies the accumulated left side. A left-deep predicate with N reportable IN expressions therefore performs O(N squared) collection work before producing the structured rejection. Please use a shared accumulator or builder while preserving encounter order and the complete diagnostic set.
**Verification:**
- **Inspection:** Trace a left-deep Boolean tree and confirm each expression node and each reported result is appended a constant number of times rather than copying an accumulated prefix.
- **Regression:** Exercise multiple UNKNOWN-sensitive nested IN expressions and verify the same complete ordered diagnostic set and rejection condition after the accumulation change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub) via reviews" <re...@spark.apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6681734655
@cloud-fan Gentle ping, as all the existing comments are addressed and the CI has passed, Could we please check this in so that i can start on the follow up on this? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4057052887
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,100 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, so it is left in the join condition, where both plans are in scope. This is only
+ * correct for an uncorrelated sub-query: the join condition of a correlated one is dropped when
+ * it is planned as an in-subquery filter, which would silently change the result, so it is
+ * reported as unsupported instead. Note that such a sub-query can be correlated only to the
+ * sub-query plan, as being correlated to the outer plan as well would require two levels of
+ * correlation, which the Analyzer rejects.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A correlated sub-query referencing both plans has to stay in the join condition, where its
+ // own join condition is lost, so reject it rather than silently return a wrong result.
+ val unsupported = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) &&
+ referencesPlan(sq, subPlan) && referencesPlan(sq, outerPlan) => sq
+ })
+ if (unsupported.nonEmpty) {
+ throw QueryCompilationErrors.unsupportedCorrelatedSubqueryInJoinConditionError(unsupported)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
Review Comment:
The targeted rejection is sufficient for this PR; I do not think the full TRUE/FALSE/NULL reconstruction needs to be folded in here. The direct null-sensitive cases are now rejected, and PullupCorrelatedPredicates runs BooleanSimplification before this rule, so NOT over AND/OR and double negation are normalized while direct NOT IN keeps the null-aware rewrite. Please track the released outer/projection issue separately.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4047869684","thread_id":"inline:4047869684","verdict_sha256":"7cf6c1474d6825b4278e346f8cc2cad4f5a46d81e53f0050f359cdd21fb70a51"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4072333034
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,14 +405,26 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
Review Comment:
Accepted in full — this was a regression I introduced when I replaced the traversal, not a
pre-existing gap, and your reading of the two lines is exactly right.
Each arm now builds its replacement through a small helper,
```scala
def replacing(sq: Expression)(replacement: => Expression): Expression = {
val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
newExpr.copyTagsFrom(sq)
newExpr
}
```
so everything an arm constructs — the `exists` attribute, the join conditions and the `Join` itself
— is built under the origin of the subquery expression being replaced, and that expression's tags
are carried over, which is what `transformDownWithPruning` did.
Your note about where the cost lands is what decided the shape of the fix: `handleUnaryNode` sends
every `UnaryNode` through this method with the default `canRewrite`, so the repair had to sit in the
arms themselves rather than in the new routing callers. The pruning check and the unchanged-node
fast path are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4139419287
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,372 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, the predicate of an `If` and the conditions of a `CaseWhen`.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) => Nil
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
Review Comment:
The collector now threads a single ArrayBuffer through the walk, preserving encounter order without repeated prefix concatenation.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4130355300","thread_id":"inline:4130355300","verdict_sha256":"15b8678aea32e2f6c0ae8bee1554ad3f1ae7e6de47363ee749ece337d2c0f9ac"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4148301436
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +159,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftSemi, joinCond, subHint)
Project(p.output, join)
case (p, Not(Exists(sub, _, _, conditions, subHint))) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftAnti, joinCond, subHint)
Project(p.output, join)
case (p, InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val (joinCond, outerPlan) = rewriteExistentialExpr(inConditions ++ conditions, p)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(inConditions ++ conditions, p, dedupSub)
Review Comment:
The premise checks out. `SizeInBytesOnlyStatsPlanVisitor.visitJoin` gives `LeftAnti | LeftSemi` an
explicit left-preserving estimate and falls through to `default(p)` for everything else, so an
`ExistenceJoin` gets the generic product estimate even though it emits exactly one row per left row
with one Boolean appended. `JoinEstimation` has no `ExistenceJoin` case either.
Two things I would like to settle before changing it, though.
It is not introduced here. `visitJoin` returns `p.left.stats` for `LeftSemi`, so an existence join
built on the outer plan — where every nested subquery went before this PR — already propagated its
inflated estimate through the enclosing semi join. This PR changes which side carries it, which is
the side broadcast eligibility is decided on, so the consequence you describe is new even though the
misestimate is not.
And the fix reaches further than this rule. Eight approved TPC-DS plans contain an `ExistenceJoin`
(q10, q35, q45 and their sf100 variants), so correcting the estimate in either statistics mode can
change those plans, and the golden files would have to be regenerated and reviewed. That is a
different kind of change from the routing fix, in a different component, with its own suites
(`JoinEstimationSuite`, `BasicStatsEstimationSuite`) and its own risk of moving estimates for plans
that are fine today.
Given that, I would rather do it as a separate JIRA, with your success criteria carried over
verbatim: a left-derived size in the size-only visitor that includes the appended Boolean width, an
explicit left-preserving estimate with left-side attribute statistics in the CBO estimator, and a
regression pinning that an `ExistenceJoin`-wrapped build side stays broadcast-eligible. Please do let know if that is acceptable? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4148351471
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
Review Comment:
Corrected. The comment now says this avoids materialising the intersection rather than claiming it
allocates nothing, and notes that the membership probes still construct wrappers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4281437494
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -184,8 +188,12 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Use EXISTS if performance matters to you.
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ // The hoisted correlated predicates may carry existential sub-queries that only the
+ // sub-query plan can evaluate, see rewriteExistentialExprInJoinCondition.
+ val (subConditions, newSub) =
+ rewriteExistentialExprInSubqueryPlan(conditions, p, dedupSub)
Review Comment:
**Non-blocking (P2):** The NOT IN arm now routes its hoisted conditions through `rewriteExistentialExprInSubqueryPlan`, which throws `NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT` for any nullable nested IN owned by the subquery plan in a value position, whether it is correlated or not. At the merge base this arm appended `conditions` unchanged to the null-aware LeftAnti condition and returned a bare `Join`, which `RewritePredicateSubquery` does not revisit. `PlanSubqueries` then evaluated an uncorrelated nested IN there through `InSubqueryExec`, which is three-valued. So a query that used to return the right rows now fails:
```sql
-- t1(a) = 1, 2, 3; t2(c1) = 1, 8, 9; t3n(col1) = 3, NULL
SELECT * FROM t1 WHERE a NOT IN (
SELECT c1 FROM t2 WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3n)) <=> false));
```
The merge base returns 2 and 3, as standard SQL does; this PR raises 0A000. `IS NULL`, `COALESCE(..., true)` and a nested NOT IN under `<=> false` behave the same way. The rejection exists because an existence join's `exists` bit cannot carry the unknown, but for an uncorrelated residual left in this anti-join condition no existence bit is built, and there is no hoisted condition to lose. The EXISTS, NOT EXISTS and IN arms are not affected: at the merge base they rewrote such an IN on the outer plan and failed with INTERNAL_ERROR. The new `NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT` tests use only an outer EXISTS, so nothing covers this arm.
**Recommended change:** Let the NOT IN arm keep uncorrelated, unknown-sensitive, subquery-plan-owned nested INs (no hoisted joinCond and no correlated descendant) as residuals in its anti-join condition, as at the merge base, instead of rejecting them. Still reject correlated ones, and still rewrite truth-only ones as now. Add SubquerySuite cases with an outer NOT IN for the value-position shapes.
**Why this works:** The helper takes an arm-supplied policy for residuals that stay three-valued. Under that policy an uncorrelated unknown-sensitive IN is declined by canRewrite and excluded from the unknown-result rejection, so it stays in the bare anti join and InSubqueryExec evaluates it, exactly as before the PR.
**Scope:** Exempt uncorrelated three-valued residuals from the unknown-result rejection in the NOT IN arm only, and test the NOT IN outer shapes.
**Compatibility:** EXISTS, NOT EXISTS and IN arms keep their current rejections.
**Risks:** If a future change makes the NOT IN arm's join revisitable, a residual left in place could be rewritten two-valued by the join-condition handler; a comment should tie the exemption to the bare-Join result.
**Constraints:** Do not exempt correlated residuals, whose hoisted condition PlanSubqueries would drop. Leave EXISTS, NOT EXISTS and IN arm behavior unchanged.
**Success:** `SELECT * FROM t1 WHERE a NOT IN (SELECT c1 FROM t2 WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3n)) <=> false))` with t3n = (3, NULL) returns 2, 3 as at the merge base. The same predicate with a correlated nested IN under an outer NOT IN is still rejected.
**Verification:**
- **Regression:** Under an outer NOT IN, uncorrelated nullable nested INs in value positions (<=> false, IS NULL, COALESCE(.., true), nested NOT IN under <=> false) return the standard SQL answer instead of an error.
- **Compatibility:** Correlated value-position nested INs under NOT IN are still rejected, and truth-only nested INs under NOT IN still return correct rows.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/JoinEstimationSuite.scala:
##########
@@ -466,6 +466,51 @@ class JoinEstimationSuite extends StatsEstimationTestBase {
}
}
+ test("existence join") {
+ // table2 (key-1-2 int, key-2-4 int): (1, 2), (2, 3), (2, 4)
+ // table3 (key-1-2 int, key-2-3 int): (1, 2), (2, 3)
+ // An existence join returns every row of its left child with one extra Boolean attribute, so
+ // its cardinality is the left child's and its width is the left child's plus that attribute.
+ // The right child's cardinality must not enter either, as the generic product estimate would
+ // have it, since that can disqualify a broadcast of an otherwise small build side.
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val join = Join(table2, table3, ExistenceJoin(exists),
+ Some(EqualTo(nameToAttr("key-2-4"), nameToAttr("key-2-3"))), JoinHint.NONE)
+
+ val expectedStats = Statistics(
+ // Three rows of the left child, each widened by the Boolean, whose default size is 1.
+ sizeInBytes = 3 * (8 + 4 * 2 + 1),
+ rowCount = Some(3),
+ // Only the left child's column stats: the right child's describe attributes that an
+ // existence join does not output.
+ attributeStats = AttributeMap(Seq(nameToColInfo("key-1-2"), nameToColInfo("key-2-4"))))
+ assert(join.stats == expectedStats)
+ }
+
+ test("existence join with an empty or unestimated right child") {
+ // Only the left child's row count enters the estimate, so a right child with no row count,
Review Comment:
**Nit (P3):** This says a right child "with no row count, or with none" must not stop the estimate, but every right child here reports a row count. `StatsTestPlan` takes a required `rowCount: BigInt`, and its `computeStats` always returns `rowCount = Some(rowCount)`, so `emptyRight` has 0 and `noStatsRight` has 2; only its column statistics are missing. If `estimateExistenceJoin`'s `rowCountsExist(join.left)` were tightened to `rowCountsExist(join.left, join.right)`, this test would stay green, and so would the BasicStatsEstimationSuite case, whose right child has rowCount 2000. A fixture whose statistics have `rowCount = None` would pin the left-only precondition that the scaladoc documents. The comment should also name the cases the test actually builds: "or with none" has no referent.
**Verification:**
- **Regression:** An existence join whose right child reports no row count is estimated from the left child exactly as for a right child with statistics.
- **Inspection:** The comment's listed cases match the fixtures the test builds.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,481 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. A nested sub-query resolves in the scope
+ * of the sub-query it is nested in, so an attribute both plans produce is one of that
+ * sub-query's own; attributing it to `outerPlan` instead would let a single shared attribute
Review Comment:
**Nit (P3):** The two paragraphs describe different alternatives to subquery-first attribution. This one says attributing a shared attribute to `outerPlan` "instead" would satisfy both sides and get an inner-only nested subquery rejected as referencing both. With the exclusive probes in `planReferences`, a swap only sets `outerPlanOnlyReference`, so `bothPlans` stays false; the claim holds only if the attribute counts for the outer plan as well. The paragraph below says the RewriteSubquerySuite fixture ends in the same error "whichever plan the shared attribute is attributed to". That holds for a swap, but not for the 'as well' reading: there the fixture's nested IN, whose joinCond `col1 = a` is non-empty, is rejected with `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY` before `buildJoin`'s dedup check. The fixture comment also uses the 'as well' reading. Please pick one alternative and state its outcome consistently in both paragraphs and in the fixture comment.
This extends an [existing discussion](https://github.com/apache/spark/pull/58656#discussion_r4257086406).
**Verification:**
- **Inspection:** Each stated outcome matches the exclusive probe order in planReferences and the error the fixture asserts.
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,540 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
Review Comment:
**Nit (P3):** The EXISTS half of this rationale no longer holds. Suppose the outer predicate were EXISTS and the new throw were removed. The join-condition handler would then reject with `UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION`, but this test asserts `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY`, so the assertion would fail rather than "hold either way". The explanation dates from when both rejections raised the same condition. NOT IN is still a reasonable choice here; only the explanation needs to change, or the EXISTS claim can be dropped.
This extends an [existing discussion](https://github.com/apache/spark/pull/58656#discussion_r4046157775).
**Verification:**
- **Inspection:** The comment's discrimination claim matches the asserted condition and the configurations the loop runs.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,481 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. A nested sub-query resolves in the scope
+ * of the sub-query it is nested in, so an attribute both plans produce is one of that
+ * sub-query's own; attributing it to `outerPlan` instead would let a single shared attribute
+ * satisfy both sides of the classification and have a nested sub-query referencing only the
+ * inner query rejected as referencing both.
+ *
+ * No plan the analyzer produces from SQL reaches this: `DeduplicateRelations` has given the
+ * sub-query its own ExprIds, and the EXISTS arms classify against the raw sub-query plan while
+ * the IN arms classify against the plan `dedupSubqueryOnSelfJoin` has already separated from the
+ * outer plan. What does reach it is the shared-attribute fixture in `RewriteSubquerySuite`,
+ * which builds the overlap directly because the analyzer is not involved there. That fixture
+ * pins the shape, not the outcome of this ordering: `dedupSubqueryOnSelfJoin`'s pre-existing
+ * conflict check reports the duplicated attribute first, so it ends in the same error whichever
+ * plan the shared attribute is attributed to.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite
+ * without rejecting the sub-query, as a sub-query referencing neither plan can be left in the
+ * join condition unharmed.
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(sq)) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(ownedBySubPlan)
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) =
+ rewriteExistentialExprWithAttrs(conditions, subPlan, ownedBySubPlan)
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ // COALESCE(x, false) maps both unknown and FALSE to FALSE, so only truth is observed. A
+ // different default does not: COALESCE(x, true) is TRUE where x is unknown and FALSE where x
+ // is FALSE.
+ case Coalesce(child +: rest) if rest.headOption.contains(Literal.FalseLiteral) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
Review Comment:
**Nit (P3):** This says a NOT directly over an IN "keeps its own null-aware rewrite, so only its operands are examined", but the arm calls `collectOrDescend(not, in, truthOnly, reported)`, which reports the NOT IN itself when it sits in a value position and may be unknown. That is why `(c1 NOT IN (SELECT col1 FROM t3n)) <=> false` is rejected, and the test asserts a reported expression starting with `(NOT `. The null-aware rewrite makes a NOT IN safe only where just truth is observed, so the comment should say the NOT IN is reported in a value position and descended into otherwise.
This extends an [existing discussion](https://github.com/apache/spark/pull/58656#discussion_r4068594158).
**Verification:**
- **Inspection:** The comment matches collectOrDescend's behavior and the rewriteExistentialExprInJoinCondition scaladoc.
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,540 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: nested IN subquery whose result can be unknown") {
+ withTempView("t1", "t2", "t2n", "t3n", "t3", "t4", "tb", "t2r", "t2rn", "t6", "t7") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // A nullable compared value, against a subquery column that is not nullable.
+ Seq(Some(1), None).toDF("c1").persist().createOrReplaceTempView("t2n")
+ Seq((1, 1), (9, 9)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ // A boolean relation, to compare the result of a nested IN subquery against.
+ Seq(false).toDF("b").persist().createOrReplaceTempView("tb")
+ // Row-valued shapes. t2r and t7 are not nullable anywhere; the nullability of t2rn and of
+ // t6 sits in the second field, so the first pair of the comparison does not expose it.
+ Seq((1, 1), (8, 8), (9, 9)).toDF("c1", "c2").persist().createOrReplaceTempView("t2r")
+ Seq((1, Option.empty[Int]), (8, Some(8))).toDF("c1", "c2n").persist()
+ .createOrReplaceTempView("t2rn")
+ Seq((1, Option.empty[Int]), (5, Some(5))).toDF("col1", "col2").persist()
+ .createOrReplaceTempView("t6")
+ Seq((1, 1), (5, 5)).toDF("col1", "col2").persist().createOrReplaceTempView("t7")
+ // The NULL matches no value of t2, so c1 IN (SELECT col1 FROM t3n) is unknown there, and
+ // c1 NOT IN (SELECT col1 FROM t3n) is never true.
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // An existence join only records whether a row matched, so its exists attribute is false
+ // where IN is unknown. That is not observable while the value feeds a predicate, but it is
+ // once it reaches anything else, so these are rejected rather than rewritten. Both
+ // configurations are checked, as the rejection must not depend on the rewrite of predicate
+ // subqueries in join conditions.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> false",
+ "(c1 IN (SELECT col1 FROM t3n)) IS NULL",
+ // COALESCE to a non-false default does tell unknown from false.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), true)",
+ "(c1 NOT IN (SELECT col1 FROM t3n)) <=> false").foreach { predicate =>
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)")
+ .collect()
+ }
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT")
+ // A NOT IN must be reported as a NOT IN, not as the IN underneath it.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.startsWith("(NOT ") == predicate.contains("NOT IN"),
+ s"unexpected subqueryExpression for $predicate: $reported")
+ }
+ }
+ }
+
+ // The ordinary case: the value of the nested IN feeds a predicate, where unknown and
+ // false cannot be told apart, so it is still rewritten even though it can be unknown.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t3n))""".stripMargin,
+ Row(1) :: Nil)
+
+ // And a nested IN that cannot be unknown is rewritten even outside a predicate.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3)) <=> false))""".stripMargin,
+ Row(1) :: Row(2) :: Row(3) :: Nil)
+
+ // Only truth is observed by `<=> true`, by an IF predicate and by a CASE WHEN
+ // condition, so those are rewritten rather than rejected. SimplifyConditionals folds
+ // the latter two into the first, so ordinary SQL reaches this position.
+ //
+ // The result branches of a conditional are consumed wherever the conditional is, so
+ // under a filter predicate they are truth-only as well. The last two cases do not
Review Comment:
**Nit (P3):** "The last two cases do not fold" points at the wrong entries. `SimplifyConditionals` folds an `If` or `CaseWhen` only when its branches are TRUE/FALSE literals, or null literals with a non-nullable condition. So all three result-branch cases stay unfolded: `IF(c1 > 0, ..., false)`, `CASE WHEN c1 > 0 THEN ... ELSE false END` and `CASE WHEN c1 > 100 THEN false ELSE ... END`. The last two entries are the ELSE-branch CASE and `COALESCE(..., false)`, which is not a conditional and has its own comment. Naming the three conditional cases would match what they exercise.
**Verification:**
- **Inspection:** The cases named as non-folding are exactly those SimplifyConditionals leaves unfolded.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) => collectOrDescend(not, in, truthOnly, reported)
+ case Not(child) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = false, reported)
+ case in: InSubquery => collectOrDescend(in, in, truthOnly, reported)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = true, reported))
+ sq.getOuterAttrs.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ case other =>
+ other.children.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ }
+
+ /**
+ * Appends `report` to `reported` if the IN sub-query `in` would lose an observable unknown, and
+ * otherwise descends into the values it compares and into its list query. The descent matters
+ * because [[InSubquery]] is a [[Predicate]] rather than a [[SubqueryExpression]], so an IN
+ * sub-query nested in the values or in the hoisted join condition of another one is only reached
+ * from here.
+ */
+ private def collectOrDescend(
+ report: Expression,
+ in: InSubquery,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = {
+ if (!truthOnly && inSubqueryMayBeUnknown(in)) {
+ reported += report
+ } else {
+ in.values.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ collectUnknownSensitiveInSubqueries(in.query, truthOnly, reported)
+ }
+ }
+
+ /**
+ * Returns true if the row comparison of the IN sub-query `in` can evaluate to unknown, that is
+ * if it can return NULL rather than only TRUE or FALSE. An existence join cannot represent that
+ * third value, see [[rewriteExistentialExprInJoinCondition]].
+ *
+ * This deliberately does not use `InSubquery.nullable`, which under
+ * `spark.sql.legacy.inSubqueryNullability` (SPARK-43413) reports only the nullability of the
+ * compared values and ignores that of the list query's columns. What is protected here is the
+ * three-valued result the rewrite would drop, which does not depend on that configuration, so
+ * the nullability is re-derived from both sides.
+ */
+ private def inSubqueryMayBeUnknown(in: InSubquery): Boolean = {
+ in.values.zip(in.query.childOutputs).exists { case (v, o) => v.nullable || o.nullable }
Review Comment:
Confirmed on 575528ff4b7: inSubqueryMayBeUnknown now zips `in.values.iterator` with `in.query.childOutputs.iterator`, so it stops at the first nullable pair.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4139416462","thread_id":"inline:4139416462","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +159,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftSemi, joinCond, subHint)
Project(p.output, join)
case (p, Not(Exists(sub, _, _, conditions, subHint))) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftAnti, joinCond, subHint)
Project(p.output, join)
case (p, InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val (joinCond, outerPlan) = rewriteExistentialExpr(inConditions ++ conditions, p)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(inConditions ++ conditions, p, dedupSub)
Review Comment:
Thanks, confirmed on 575528ff4b7. Both SizeInBytesOnlyStatsPlanVisitor and JoinEstimation now estimate an ExistenceJoin from its left child with the Boolean width added, the statistics suites pin both modes, and the SubquerySuite test keeps the enclosing semi join broadcast.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4139416473","thread_id":"inline:4139416473","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) => collectOrDescend(not, in, truthOnly, reported)
+ case Not(child) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = false, reported)
+ case in: InSubquery => collectOrDescend(in, in, truthOnly, reported)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = true, reported))
+ sq.getOuterAttrs.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ case other =>
+ other.children.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ }
+
+ /**
+ * Appends `report` to `reported` if the IN sub-query `in` would lose an observable unknown, and
+ * otherwise descends into the values it compares and into its list query. The descent matters
+ * because [[InSubquery]] is a [[Predicate]] rather than a [[SubqueryExpression]], so an IN
+ * sub-query nested in the values or in the hoisted join condition of another one is only reached
+ * from here.
+ */
+ private def collectOrDescend(
+ report: Expression,
+ in: InSubquery,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = {
+ if (!truthOnly && inSubqueryMayBeUnknown(in)) {
+ reported += report
+ } else {
+ in.values.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ collectUnknownSensitiveInSubqueries(in.query, truthOnly, reported)
+ }
+ }
+
+ /**
+ * Returns true if the row comparison of the IN sub-query `in` can evaluate to unknown, that is
+ * if it can return NULL rather than only TRUE or FALSE. An existence join cannot represent that
+ * third value, see [[rewriteExistentialExprInJoinCondition]].
+ *
+ * This deliberately does not use `InSubquery.nullable`, which under
+ * `spark.sql.legacy.inSubqueryNullability` (SPARK-43413) reports only the nullability of the
+ * compared values and ignores that of the list query's columns. What is protected here is the
+ * three-valued result the rewrite would drop, which does not depend on that configuration, so
+ * the nullability is re-derived from both sides.
+ */
+ private def inSubqueryMayBeUnknown(in: InSubquery): Boolean = {
+ in.values.zip(in.query.childOutputs).exists { case (v, o) => v.nullable || o.nullable }
+ }
+
+ /**
+ * Returns true if `e` is an existential sub-query expression that still carries a correlated
+ * condition, that is one that [[PullupCorrelatedPredicates]] hoisted into it and that would be
+ * lost were the sub-query left in a join condition.
+ *
+ * This is decided from that condition rather than from `isCorrelated`, which reports the outer
Review Comment:
Confirmed: the scaladoc now describes isCorrelated as `outerAttrs.nonEmpty`, which makes it clear why the helper reads the surviving join condition instead.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4139416480","thread_id":"inline:4139416480","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
Review Comment:
Confirmed: the scaladoc now names both paths, decorrelation disabled and an uncorrelated IN with optimizeUncorrelatedInSubqueriesInJoinCondition.enabled disabled.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4139416485","thread_id":"inline:4139416485","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4274291543
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,426 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: nested IN subquery whose result can be unknown") {
+ withTempView("t1", "t2", "t2n", "t3n", "t3", "t4", "tb") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // A nullable compared value, against a subquery column that is not nullable.
+ Seq(Some(1), None).toDF("c1").persist().createOrReplaceTempView("t2n")
+ Seq((1, 1), (9, 9)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ // A boolean relation, to compare the result of a nested IN subquery against.
+ Seq(false).toDF("b").persist().createOrReplaceTempView("tb")
+ // The NULL matches no value of t2, so c1 IN (SELECT col1 FROM t3n) is unknown there, and
+ // c1 NOT IN (SELECT col1 FROM t3n) is never true.
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // An existence join only records whether a row matched, so its exists attribute is false
+ // where IN is unknown. That is not observable while the value feeds a predicate, but it is
+ // once it reaches anything else, so these are rejected rather than rewritten. Both
+ // configurations are checked, as the rejection must not depend on the rewrite of predicate
+ // subqueries in join conditions.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> false",
+ "(c1 IN (SELECT col1 FROM t3n)) IS NULL",
+ // COALESCE to a non-false default does tell unknown from false.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), true)",
+ "(c1 NOT IN (SELECT col1 FROM t3n)) <=> false").foreach { predicate =>
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)")
+ .collect()
+ }
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT")
+ // A NOT IN must be reported as a NOT IN, not as the IN underneath it.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.startsWith("(NOT ") == predicate.contains("NOT IN"),
+ s"unexpected subqueryExpression for $predicate: $reported")
+ }
+ }
+ }
+
+ // The ordinary case: the value of the nested IN feeds a predicate, where unknown and
+ // false cannot be told apart, so it is still rewritten even though it can be unknown.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t3n))""".stripMargin,
+ Row(1) :: Nil)
+
+ // And a nested IN that cannot be unknown is rewritten even outside a predicate.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3)) <=> false))""".stripMargin,
+ Row(1) :: Row(2) :: Row(3) :: Nil)
+
+ // Only truth is observed by `<=> true`, by an IF predicate and by a CASE WHEN
+ // condition, so those are rewritten rather than rejected. SimplifyConditionals folds
+ // the latter two into the first, so ordinary SQL reaches this position.
+ //
+ // The result branches of a conditional are consumed wherever the conditional is, so
+ // under a filter predicate they are truth-only as well. The last two cases do not
+ // fold, so they exercise that inheritance rather than the rewrites above.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> true",
+ "IF(c1 IN (SELECT col1 FROM t3n), true, false)",
+ "CASE WHEN c1 IN (SELECT col1 FROM t3n) THEN true ELSE false END",
+ "IF(c1 > 0, c1 IN (SELECT col1 FROM t3n), false)",
+ "CASE WHEN c1 > 0 THEN c1 IN (SELECT col1 FROM t3n) ELSE false END",
+ "CASE WHEN c1 > 100 THEN false ELSE c1 IN (SELECT col1 FROM t3n) END",
+ // COALESCE to false maps unknown and false alike, so only truth is observed.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), false)")
+ .foreach { predicate =>
+ checkAnswerAndPlan(
+ s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)",
+ Row(1) :: Nil)
+ }
+
+ // The unknown can come from the compared value rather than from the subquery column.
+ // spark.sql.legacy.inSubqueryNullability makes InSubquery.nullable ignore the
+ // nullability of the list query columns, but it always keeps that of the compared
+ // values, and the guard re-derives both, so this case is rejected under either value.
+ Seq("true", "false").foreach { legacyNullability =>
Review Comment:
The coverage gap is real — you are right that the existing loop uses nullable compared values, so both flag values report them nullable and the loop would not notice a regression on the RHS-derived side. I have added the case.
Your premise about what it would show does not hold, though, and I would have asserted the wrong thing if I had followed the recommendation. With a non-nullable compared value (`t2.c1`) and a nullable list output (`t3n.col1`), the two settings legitimately disagree:
| `spark.sql.legacy.inSubqueryNullability` | outcome |
|---|---|
| `false` (default) | `NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT`, subqueryExpression `(t2.c1 IN (listquery()))` |
| `true` | accepted, returns `1` — **the correct answer** |
Under legacy mode the query is not an UNKNOWN collapsed to FALSE. Because `InSubquery.nullable` is false there, `SimplifyBinaryComparison` turns `x <=> false` into `NOT x`, and a `NOT` directly over an IN takes the null-aware rewrite:
```
Join LeftSemi, ((value#103 = c1#114) OR NOT exists#127)
+- Join ExistenceJoin(exists#127), ((c1#114 = col1#121) OR isnull((c1#114 = col1#121)))
```
`exists` is TRUE where the comparison is unknown, so `NOT exists` is FALSE there — which is exactly what `unknown <=> false` is. DuckDB returns `1` for the same data, so legacy mode is right here, by a different route. The guard is not reached because there is no longer an `EqualNullSafe` to reach, not because it failed to re-derive nullability.
So the test now pins the rejection under the default and the *correct answer and plan* under legacy mode, with a comment explaining why they differ. Asserting rejection under both, as recommended, would have been asserting a bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4414366426
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,625 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. With EXISTS the rewrite returns a Project over the join, which that handling
+ // reaches under the default configuration and rejects with
+ // UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION instead, so the query would still
+ // fail but on a different condition than the one asserted here.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: existential subquery carried by a rewritten nested subquery") {
+ // The nested IN over t4 is owned by the subquery plan, so it is rewritten into an existence
+ // join and the condition PullupCorrelatedPredicates hoisted into it is grafted onto that
+ // join. That condition carries a third-level IN correlated to t4, which references neither
+ // the outer nor the subquery plan and so is routed by neither of the guards above. It has to
+ // be rewritten against the inputs of the join just built: left in place, PlanSubqueries plans
+ // it as an in-subquery filter and drops its own correlated predicate col1 = k, which returns
+ // every row of t1 instead of the one row where the correlation holds.
+ withTempView("t1", "t2", "t4", "t5") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ // No k equals any c1, so only the third-level IN can satisfy the hoisted disjunction, and
+ // its correlated predicate col1 = k is false for every row of t4: dropping it makes the
+ // level-2 IN hold for c1 = 1 and returns 1, 2 and 3.
+ Seq((1, 100), (9, 7)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(1).toDF("col1").persist().createOrReplaceTempView("t5")
+ val query =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t4
+ | WHERE k = c1 OR col1 IN (SELECT col1 FROM t5 WHERE col1 = k)))""".stripMargin
+ // The routing must not depend on the rewrite of predicate subqueries in join conditions,
+ // which is what used to make these two configurations disagree: it reached the grafted
+ // condition only when enabled.
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswerAndPlan(query, Row(1) :: Nil)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: NOT IN keeps an uncorrelated three-valued residual") {
+ // The NOT IN rewrite returns a bare Join, which this rule does not offer to its own handling
+ // of predicate subqueries in join conditions, so an uncorrelated nested IN left in the
+ // anti-join condition stays there and InSubqueryExec evaluates it three-valued. No existence
+ // join is built for it and it has no hoisted condition to lose, so it must not be rejected:
+ // these queries returned the standard SQL answer before SPARK-59351 and must keep doing so.
+ withTempView("t1", "t2", "t3n", "t4n") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ // The NULL matches no c1, so c1 IN (SELECT col1 FROM t3n) is unknown for every row of t2.
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // Nullable col1 again, but correlated through k = c1.
+ Seq((Some(3), 1), (None, 8)).toDF("col1", "k").persist().createOrReplaceTempView("t4n")
+
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> false",
+ "(c1 IN (SELECT col1 FROM t3n)) IS NULL",
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), true)",
+ "(c1 NOT IN (SELECT col1 FROM t3n)) <=> false").foreach { predicate =>
+ // Every such predicate is FALSE for every row of t2, so the subquery reduces to a = c1
+ // and only 2 and 3 survive the NOT IN. Rejecting instead, or rewriting the nested IN into
+ // a two-valued exists, would change this.
+ checkAnswer(
+ sql(s"SELECT * FROM t1 WHERE a NOT IN (SELECT c1 FROM t2 WHERE a = c1 OR $predicate)"),
Review Comment:
Confirmed: with c1 > a hoisted, each residual case now tells the three-valued residual apart from a two-valued rewrite, and the truth-only case goes through checkAnswerAndPlan.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4346076727","thread_id":"inline:4346076727","verdict_sha256":"596d6ef5f193363401f54af770e62e1e49b3bfd27c84efadb74d430a0c108603"} -->
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,625 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. With EXISTS the rewrite returns a Project over the join, which that handling
+ // reaches under the default configuration and rejects with
+ // UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION instead, so the query would still
+ // fail but on a different condition than the one asserted here.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: existential subquery carried by a rewritten nested subquery") {
Review Comment:
Confirmed: the new test covers the Filter-predicate, projection and join-condition routes with a correlated nested IN whose predicate decides the answer.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4346076746","thread_id":"inline:4346076746","verdict_sha256":"596d6ef5f193363401f54af770e62e1e49b3bfd27c84efadb74d430a0c108603"} -->
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,625 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. With EXISTS the rewrite returns a Project over the join, which that handling
Review Comment:
Confirmed: the comment now states the EXISTS outcome as a counterfactual and matches the asserted condition.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4346076758","thread_id":"inline:4346076758","verdict_sha256":"596d6ef5f193363401f54af770e62e1e49b3bfd27c84efadb74d430a0c108603"} -->
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala:
##########
@@ -161,4 +163,124 @@ class RewriteSubquerySuite extends PlanTest {
val existsExpected = LocalRelation(existsQuery.analyze.output).analyze
comparePlans(existsOptimized, existsExpected)
}
+
+ test("SPARK-59351: nested subquery of an EXISTS whose plan shares attributes with the outer " +
+ "plan") {
+ // The EXISTS arms route the nested subquery against the raw subquery plan, while the IN arms
+ // route it against the plan dedupSubqueryOnSelfJoin has already separated from the outer
+ // plan. A self-join can leave an ExprId shared between the two plans for the optimizer
+ // (SPARK-21835), and the analyzer is not involved here, so build that state directly: `a` is
+ // the same attribute in the outer and in the subquery plan.
+ val a = $"a".int
+ val outer = LocalRelation(a, $"b".int)
+ val sub = LocalRelation(a)
+ val nested = LocalRelation($"col1".int)
+ assert(outer.outputSet.intersect(sub.outputSet).nonEmpty,
+ "the two plans must share an attribute for this test to mean anything")
+
+ // The nested subquery references `a`, which only the subquery plan is meant to give it, and
+ // is correlated to it, so it must be rewritten against the subquery plan. Classifying `a` as
Review Comment:
Confirmed: the fixture comment now describes the swap, matching the planReferences scaladoc.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4346076766","thread_id":"inline:4346076766","verdict_sha256":"596d6ef5f193363401f54af770e62e1e49b3bfd27c84efadb74d430a0c108603"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothInputs.toSeq)
+ }
+ val unknownResult = carried.filter(sq => referencesOneInput(sq) && observesUnknown(sq))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult.toSeq)
+ }
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ carried.filter(sq => referencesOneInput(sq) && isInput(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) = rewriteExistentialExprWithAttrs(Seq(sq), p)
+ newCondition = newCondition.transform {
+ case e if e.fastEquals(sq) => rewritten.get
+ }
+ newInput
+ }
+ }
+ val newLeft = route(join.left, referencesInput(_, join.left))
+ val newRight = route(join.right, referencesInput(_, join.right))
+ if (newLeft.fastEquals(join.left) && newRight.fastEquals(join.right)) {
+ join
+ } else {
+ // Drop the `exists` attributes the nested existence joins introduced, keeping this join's
+ // own output, which already carries its own.
+ Project(join.output, join.copy(
+ left = newLeft, right = newRight, condition = Some(newCondition)))
+ }
+ }
+
+ /**
+ * Collects the existential sub-queries carried by the condition of an existence join, as the
+ * units [[rewriteExistentialExprWithAttrs]] rewrites: a NOT IN is collected as the [[Not]] over
+ * the [[InSubquery]], so the rewrite takes its null-aware arm and the unknown-result collector,
+ * which reports the same node, can be compared against it.
+ *
+ * A collected sub-query is not descended into. One nested below it is reached when it is
+ * rewritten, since the join built for it goes through the routing in turn, and one below a
+ * sub-query left in place is left with it, as everywhere else in this rule.
+ */
+ private def collectCarriedExistentialExprs(
+ expr: Expression,
+ collected: ArrayBuffer[Expression]): Unit = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return
+ }
+ expr match {
+ case sq @ Not(_: InSubquery) => collected += sq
+ case sq: Exists => collected += sq
+ case sq: InSubquery => collected += sq
+ case other => other.children.foreach(collectCarriedExistentialExprs(_, collected))
+ }
+ }
+
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan = rewriteCarriedExistentialExprs(
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint))
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint))
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(
+ Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint))
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. A nested sub-query resolves in the scope
+ * of the sub-query it is nested in, so an attribute both plans produce is one of that
+ * sub-query's own.
+ *
+ * The probes are exclusive, so swapping their order would attribute such an attribute to
+ * `outerPlan` alone: `subPlanOnly` and `bothPlans` would both be false, and a nested sub-query
+ * whose only reference is the shared attribute would be declined and left in the join condition
+ * rather than routed to the sub-query plan that can evaluate it.
+ *
+ * No plan the analyzer produces from SQL reaches this: `DeduplicateRelations` has given the
+ * sub-query its own ExprIds, and the EXISTS arms classify against the raw sub-query plan while
+ * the IN arms classify against the plan `dedupSubqueryOnSelfJoin` has already separated from the
+ * outer plan. What does reach it is the shared-attribute fixture in `RewriteSubquerySuite`,
+ * which builds the overlap directly because the analyzer is not involved there. That fixture
+ * pins the shape, not the outcome of this ordering: under either order it ends in the same
+ * error, since `dedupSubqueryOnSelfJoin`'s pre-existing conflict check reports the duplicated
+ * attribute that the hoisted condition mentions.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite here.
+ * That does not leave the sub-query unexamined: one carried by a nested sub-query that is
+ * rewritten is routed against the inputs of the join built for it, see
+ * [[rewriteCarriedExistentialExprs]].
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
Review Comment:
Confirmed: the scaladoc now limits the rejection to sub-queries the sub-query plan owns and points the outer-only case at SPARK-59862.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4369134663","thread_id":"inline:4369134663","verdict_sha256":"596d6ef5f193363401f54af770e62e1e49b3bfd27c84efadb74d430a0c108603"} -->
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,749 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. The EXISTS form of this query fails on the same condition, raised by the same
+ // rejection in rewriteExistentialExprInSubqueryPlan before any join exists; were that
+ // rejection removed, the EXISTS form would instead reach the join-condition handling, which
+ // rejects it with UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION under the default
+ // configuration, so only the NOT IN form pins this rejection on its own.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: existential subquery carried by a rewritten nested subquery") {
+ // The nested IN over t4 is owned by the subquery plan, so it is rewritten into an existence
+ // join and the condition PullupCorrelatedPredicates hoisted into it is grafted onto that
+ // join. That condition carries a third-level IN correlated to t4, which references neither
+ // the outer nor the subquery plan and so is routed by neither of the guards above. It has to
+ // be rewritten against the inputs of the join just built: left in place, PlanSubqueries plans
+ // it as an in-subquery filter and drops its own correlated predicate col1 = k, which returns
+ // every row of t1 instead of the one row where the correlation holds.
+ withTempView("t1", "t2", "t4", "t5") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ // No k equals any c1, so only the third-level IN can satisfy the hoisted disjunction, and
+ // its correlated predicate col1 = k is false for every row of t4: dropping it makes the
+ // level-2 IN hold for c1 = 1 and returns 1, 2 and 3.
+ Seq((1, 100), (9, 7)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(1).toDF("col1").persist().createOrReplaceTempView("t5")
+ val query =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t4
+ | WHERE k = c1 OR col1 IN (SELECT col1 FROM t5 WHERE col1 = k)))""".stripMargin
+ // The routing must not depend on the rewrite of predicate subqueries in join conditions,
+ // which is what used to make these two configurations disagree: it reached the grafted
+ // condition only when enabled.
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswerAndPlan(query, Row(1) :: Nil)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: a carried subquery is classified as on the subquery-plan route") {
+ // The carried router applies the same rules as rewriteExistentialExprInSubqueryPlan, against
+ // the inputs of the existence join rather than the outer and the subquery plan.
+ withTempView("t1", "t2", "t4", "t5", "t3n", "t5n") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((9, 7)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(Some(3), None).toDF("x").persist().createOrReplaceTempView("t3n")
+ Seq((Some(3), 7)).toDF("x", "k").persist().createOrReplaceTempView("t5n")
+
+ // A carried NOT IN is collected as the Not over the IN, so the rewrite takes its null-aware
+ // arm. Taking the positive-IN arm instead leaves NOT exists over a condition that ignores
+ // NULL comparisons: 5 NOT IN (NULL) is unknown, so the t4 row is excluded and the answer is
Review Comment:
Confirmed: the comment now names n4's (5, 100) and n1's 1 and 2, which match the case's own fixtures.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4369134666","thread_id":"inline:4369134666","verdict_sha256":"596d6ef5f193363401f54af770e62e1e49b3bfd27c84efadb74d430a0c108603"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4356976743
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,625 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. With EXISTS the rewrite returns a Project over the join, which that handling
+ // reaches under the default configuration and rejects with
+ // UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION instead, so the query would still
+ // fail but on a different condition than the one asserted here.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: existential subquery carried by a rewritten nested subquery") {
Review Comment:
Right -- the router runs on every existence join the rule builds, and only the Filter-conjunct route was covered. Added `carried subqueries are routed on every route that builds an existence join`, using a correlated nested IN whose predicate decides the answer: it is false for every `c1` of t2, while the same IN without `k = c1` holds for `c1 = 1`, so losing it turns every answer into every row.
* **Filter-predicate arm** -- `WHERE a > 100 OR EXISTS (...)`, where the EXISTS is not a conjunct of its own: expects `1`, both configurations.
* **`handleUnaryNode`** -- `SELECT a, EXISTS (...) AS e FROM t1`: expects `(1, true), (2, false), (3, false)`, both configurations.
* **join-condition arm** -- `FROM t1 JOIN t6 ON t1.a = t6.b AND EXISTS (...)`: expects `1`. Only under the enabled configuration, since with it disabled that arm does not run at all and the sub-query stays in the join condition, which is pre-existing behaviour this PR does not change; the test says so.
Checked as you asked: with the carried collection disabled, the new test fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4356981733
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala:
##########
@@ -161,4 +163,124 @@ class RewriteSubquerySuite extends PlanTest {
val existsExpected = LocalRelation(existsQuery.analyze.output).analyze
comparePlans(existsOptimized, existsExpected)
}
+
+ test("SPARK-59351: nested subquery of an EXISTS whose plan shares attributes with the outer " +
+ "plan") {
+ // The EXISTS arms route the nested subquery against the raw subquery plan, while the IN arms
+ // route it against the plan dedupSubqueryOnSelfJoin has already separated from the outer
+ // plan. A self-join can leave an ExprId shared between the two plans for the optimizer
+ // (SPARK-21835), and the analyzer is not involved here, so build that state directly: `a` is
+ // the same attribute in the outer and in the subquery plan.
+ val a = $"a".int
+ val outer = LocalRelation(a, $"b".int)
+ val sub = LocalRelation(a)
+ val nested = LocalRelation($"col1".int)
+ assert(outer.outputSet.intersect(sub.outputSet).nonEmpty,
+ "the two plans must share an attribute for this test to mean anything")
+
+ // The nested subquery references `a`, which only the subquery plan is meant to give it, and
+ // is correlated to it, so it must be rewritten against the subquery plan. Classifying `a` as
Review Comment:
Fixed, and apologies -- last round I aligned the paragraph below this one and left this sentence, which is why the thread had to be reopened. It now states the swap and its outcome
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub) via reviews" <re...@spark.apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6596139225
@cloud-fan Gentle Ping!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4133806036
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,372 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, the predicate of an `If` and the conditions of a `CaseWhen`.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) => Nil
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
+ unknownSensitiveInSubqueries(right, truthOnly)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
+ unknownSensitiveInSubqueries(right, truthOnly)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here. The
+ // same holds for the branch conditions of a conditional.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ unknownSensitiveInSubqueries(left, truthOnly = true)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ unknownSensitiveInSubqueries(right, truthOnly = true)
+ case If(predicate, trueValue, falseValue) =>
+ unknownSensitiveInSubqueries(predicate, truthOnly = true) ++
+ unknownSensitiveInSubqueries(trueValue, truthOnly = false) ++
+ unknownSensitiveInSubqueries(falseValue, truthOnly = false)
+ case CaseWhen(branches, elseValue) =>
+ branches.flatMap { case (condition, value) =>
+ unknownSensitiveInSubqueries(condition, truthOnly = true) ++
+ unknownSensitiveInSubqueries(value, truthOnly = false)
+ } ++ elseValue.toSeq.flatMap(unknownSensitiveInSubqueries(_, truthOnly = false))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) =>
+ collectOrDescend(not, in, truthOnly)
+ case Not(child) => unknownSensitiveInSubqueries(child, truthOnly = false)
+ case in: InSubquery =>
+ collectOrDescend(in, in, truthOnly)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.flatMap(unknownSensitiveInSubqueries(_, truthOnly = true)) ++
+ sq.getOuterAttrs.flatMap(unknownSensitiveInSubqueries(_, truthOnly = false))
+ case other =>
+ other.children.flatMap(unknownSensitiveInSubqueries(_, truthOnly = false))
+ }
+
+ /**
+ * Collects `reported` if the IN sub-query `in` would lose an observable unknown, and otherwise
+ * descends into the values it compares and into its list query. The descent matters because
+ * [[InSubquery]] is a [[Predicate]] rather than a [[SubqueryExpression]], so an IN sub-query
+ * nested in the values or in the hoisted join condition of another one is only reached from
+ * here.
+ */
+ private def collectOrDescend(
+ reported: Expression,
+ in: InSubquery,
+ truthOnly: Boolean): Seq[Expression] = {
+ if (!truthOnly && inSubqueryMayBeUnknown(in)) {
+ Seq(reported)
+ } else {
+ in.values.flatMap(unknownSensitiveInSubqueries(_, truthOnly = false)) ++
+ unknownSensitiveInSubqueries(in.query, truthOnly)
+ }
+ }
+
+ /**
+ * Returns true if the row comparison of the IN sub-query `in` can evaluate to unknown, that is
+ * if it can return NULL rather than only TRUE or FALSE. An existence join cannot represent that
+ * third value, see [[rewriteExistentialExprInJoinCondition]].
+ *
+ * This deliberately does not use `InSubquery.nullable`, which returns false for either side
Review Comment:
Fixed. The walk now appends into a single `ArrayBuffer` passed down the recursion instead of
concatenating the returned prefixes at every `And` or `Or`, so each node and each reported
expression is appended once. `unknownSensitiveInSubqueries` keeps its signature and returns the
buffer, so encounter order and the complete diagnostic set are unchanged, which the multi-subquery
cases in the suite still assert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4148371242
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
Review Comment:
Fair, and the gap was real: dropping either would have left every existing assertion green. There is
now a test in `RewriteSubquerySuite` that gives the sub-query expression a distinctive `Origin` and a
`TreeNodeTag`, runs the rule, and asserts the replacement carries both. It fails if `withOrigin` or
`copyTagsFrom` is removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4414363509
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +417,649 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, a correlated IN or NOT IN is planned by `PlanSubqueries`
Review Comment:
**Nit (P3):** 'Nothing else reaches it' holds only with `decorrelatePredicateSubqueriesInJoinPredicate.enabled` off. Under the default, the top-down pass visits the existence joins this rule has just built in the `case j: Join` arm, which rewrites a carried correlated IN or EXISTS against the input it references. The PR's own test comment says the same ('it reached the grafted condition only when enabled'). The next paragraph's 'for a join the user wrote' has the same problem. The scaladoc would match the code if it said that the hoisted condition is dropped only with that setting off, and that under the default the router adds config independence and the unknown-result guard. The 'only this router reaches it' comment in the carried-routes test needs the same change.
This extends an [existing discussion](https://github.com/apache/spark/pull/58656#discussion_r4369134661).
**Verification:**
- **Inspection:** The router's scaladoc and the carried-route test comment both state that, under the default decorrelatePredicateSubqueriesInJoinPredicate setting, the join-condition handling also reaches the existence joins this rule builds.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +417,649 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, a correlated IN or NOT IN is planned by `PlanSubqueries`
+ * as an in-subquery filter that drops its own hoisted condition, which returns extra rows, and
+ * a correlated EXISTS reaches execution as an unevaluable `Exists` and fails.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ *
+ * The unknown-result rejection applies only when `rejectsUnknownResult` is set, which is the
+ * case for the existence joins built on the sub-query-plan route, where the same shapes failed
+ * with an internal error. On the routes that predate this change an uncorrelated IN whose
+ * unknown result is observable is left where it is, as it was, and a correlated one is routed
+ * without the rejection. Their two-valued handling is tracked by SPARK-59862.
+ *
+ * A NOT IN is routed before a bare IN, since the substitution of a unit replaces every
+ * occurrence of it, including one inside a `Not` over the same instance.
+ */
+ private def rewriteCarriedExistentialExprs(
+ join: Join,
+ rejectsUnknownResult: Boolean): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothInputs.toSeq)
+ }
+ // Only the sub-query-plan route rejects. An existence join built on another route answered
+ // such a sub-query before this routing existed, three-valued where it was left in place, and
+ // that is left to SPARK-59862: see `rejectsUnknownResult`.
+ def uncorrelated(sq: Expression): Boolean =
+ !hasCorrelatedCondition(sq) && !carriesCorrelatedSubquery(negatedIn(sq))
+ if (rejectsUnknownResult) {
+ val unknownResult = carried.filter(sq => referencesOneInput(sq) && observesUnknown(sq))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult.toSeq)
+ }
+ }
+ // A NOT IN unit goes before a bare IN: the substitution of a unit replaces every occurrence
+ // of it in the condition, so a bare IN routed first would also replace the one inside a
+ // `Not` over the same instance, and the null-aware join built for that `Not` would be left
+ // unreferenced. The sort is stable, so the order within each kind is the order collected.
+ val routed = carried.sortBy {
+ case Not(_: InSubquery) => 0
+ case _ => 1
+ }.filterNot(sq => !rejectsUnknownResult && observesUnknown(sq) && uncorrelated(sq))
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ routed.filter(sq => referencesOneInput(sq) && isInput(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) =
+ rewriteExistentialExprWithAttrs(Seq(sq), p, rejectsUnknownResult = rejectsUnknownResult)
+ newCondition = newCondition.transform {
+ case e if e.fastEquals(sq) => rewritten.get
+ }
+ newInput
+ }
+ }
+ val newLeft = route(join.left, referencesInput(_, join.left))
+ val newRight = route(join.right, referencesInput(_, join.right))
+ if (newLeft.fastEquals(join.left) && newRight.fastEquals(join.right)) {
+ join
+ } else {
+ // Drop the `exists` attributes the nested existence joins introduced, keeping this join's
+ // own output, which already carries its own.
+ Project(join.output, join.copy(
+ left = newLeft, right = newRight, condition = Some(newCondition)))
+ }
+ }
+
+ /**
+ * Collects the existential sub-queries carried by the condition of an existence join, as the
+ * units [[rewriteExistentialExprWithAttrs]] rewrites: a NOT IN is collected as the [[Not]] over
+ * the [[InSubquery]], so the rewrite takes its null-aware arm and the unknown-result collector,
+ * which reports the same node, can be compared against it.
+ *
+ * A collected sub-query is not descended into. One nested below it is reached when it is
+ * rewritten, since the join built for it goes through the routing in turn, and one below a
+ * sub-query left in place is left with it, as everywhere else in this rule.
+ */
+ private def collectCarriedExistentialExprs(
+ expr: Expression,
+ collected: ArrayBuffer[Expression]): Unit = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return
+ }
+ expr match {
+ case sq @ Not(_: InSubquery) => collected += sq
+ case sq: Exists => collected += sq
+ case sq: InSubquery => collected += sq
+ case other => other.children.foreach(collectCarriedExistentialExprs(_, collected))
+ }
+ }
+
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true,
+ rejectsUnknownResult: Boolean = false): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan = rewriteCarriedExistentialExprs(
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint),
+ rejectsUnknownResult)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint), rejectsUnknownResult)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(
+ Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint), rejectsUnknownResult)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. A nested sub-query resolves in the scope
+ * of the sub-query it is nested in, so an attribute both plans produce is one of that
+ * sub-query's own.
+ *
+ * The probes are exclusive, so swapping their order would attribute such an attribute to
+ * `outerPlan` alone: `subPlanOnly` and `bothPlans` would both be false, and a nested sub-query
+ * whose only reference is the shared attribute would be declined and left in the join condition
+ * rather than routed to the sub-query plan that can evaluate it.
+ *
+ * No plan the analyzer produces from SQL reaches this: `DeduplicateRelations` has given the
+ * sub-query its own ExprIds, and the EXISTS arms classify against the raw sub-query plan while
+ * the IN arms classify against the plan `dedupSubqueryOnSelfJoin` has already separated from the
+ * outer plan. What does reach it is the shared-attribute fixture in `RewriteSubquerySuite`,
+ * which builds the overlap directly because the analyzer is not involved there. That fixture
+ * pins the shape, not the outcome of this ordering: under either order it ends in the same
+ * error, since `dedupSubqueryOnSelfJoin`'s pre-existing conflict check reports the duplicated
+ * attribute that the hoisted condition mentions.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite here.
+ * That does not leave the sub-query unexamined: one carried by a nested sub-query that is
+ * rewritten is routed against the inputs of the join built for it, see
+ * [[rewriteCarriedExistentialExprs]].
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * The rejection covers the sub-queries the sub-query plan owns. An IN sub-query that only
+ * references the outer plan is still rewritten two-valued, as it was before SPARK-59351, which
+ * SPARK-59862 tracks.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan,
+ keepsThreeValuedResiduals: Boolean = false): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(sq)) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above. Unless the caller keeps such
+ // a sub-query as a residual, in which case no existence join is built for it and there is
+ // nothing to reject: see `keepsThreeValuedResiduals`.
+ val unknownSensitive = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(ownedBySubPlan)
+ val (residual, unknownResult) = unknownSensitive.partition(sq =>
+ keepsThreeValuedResiduals && !hasCorrelatedCondition(sq) && !carriesCorrelatedSubquery(sq))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ def canRewrite(sq: Expression): Boolean =
+ ownedBySubPlan(sq) && !residual.exists(_.fastEquals(sq))
+ val (newCond, newSubPlan, _) =
+ rewriteExistentialExprWithAttrs(conditions, subPlan, canRewrite, rejectsUnknownResult = true)
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ // COALESCE(x, false) maps both unknown and FALSE to FALSE, so only truth is observed. A
+ // different default does not: COALESCE(x, true) is TRUE where x is unknown and FALSE where x
+ // is FALSE.
+ case Coalesce(child +: rest) if rest.headOption.contains(Literal.FalseLiteral) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query is handled as that IN is, by `collectOrDescend`: it is
+ // reported in a value position, where the unknown its null-aware rewrite preserves would be
Review Comment:
**Nit (P3):** The null-aware rewrite does not preserve the unknown. Its condition `c OR isnull(c)` makes `exists` TRUE where the comparison is unknown, so `NOT exists` is FALSE there. That is why the `rewriteExistentialExprInJoinCondition` scaladoc calls it 'equally two-valued'. For `(c1 NOT IN (...)) <=> false` with a non-matching NULL, SQL gives FALSE but the rewrite would give TRUE. What a value position would observe is the FALSE the rewrite yields in place of unknown, so the comment could say 'where the FALSE its null-aware rewrite yields in place of unknown would be observable'.
This extends an [existing discussion](https://github.com/apache/spark/pull/58656#discussion_r4281437515).
**Verification:**
- **Inspection:** The collector's NOT IN comment states that the null-aware rewrite turns an unknown NOT IN into FALSE, and that this is why a value-position NOT IN is reported.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +417,649 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, a correlated IN or NOT IN is planned by `PlanSubqueries`
+ * as an in-subquery filter that drops its own hoisted condition, which returns extra rows, and
+ * a correlated EXISTS reaches execution as an unevaluable `Exists` and fails.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ *
+ * The unknown-result rejection applies only when `rejectsUnknownResult` is set, which is the
+ * case for the existence joins built on the sub-query-plan route, where the same shapes failed
+ * with an internal error. On the routes that predate this change an uncorrelated IN whose
+ * unknown result is observable is left where it is, as it was, and a correlated one is routed
+ * without the rejection. Their two-valued handling is tracked by SPARK-59862.
+ *
+ * A NOT IN is routed before a bare IN, since the substitution of a unit replaces every
+ * occurrence of it, including one inside a `Not` over the same instance.
+ */
+ private def rewriteCarriedExistentialExprs(
+ join: Join,
+ rejectsUnknownResult: Boolean): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothInputs.toSeq)
+ }
+ // Only the sub-query-plan route rejects. An existence join built on another route answered
+ // such a sub-query before this routing existed, three-valued where it was left in place, and
+ // that is left to SPARK-59862: see `rejectsUnknownResult`.
+ def uncorrelated(sq: Expression): Boolean =
+ !hasCorrelatedCondition(sq) && !carriesCorrelatedSubquery(negatedIn(sq))
+ if (rejectsUnknownResult) {
+ val unknownResult = carried.filter(sq => referencesOneInput(sq) && observesUnknown(sq))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult.toSeq)
+ }
+ }
+ // A NOT IN unit goes before a bare IN: the substitution of a unit replaces every occurrence
+ // of it in the condition, so a bare IN routed first would also replace the one inside a
+ // `Not` over the same instance, and the null-aware join built for that `Not` would be left
+ // unreferenced. The sort is stable, so the order within each kind is the order collected.
+ val routed = carried.sortBy {
+ case Not(_: InSubquery) => 0
+ case _ => 1
+ }.filterNot(sq => !rejectsUnknownResult && observesUnknown(sq) && uncorrelated(sq))
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ routed.filter(sq => referencesOneInput(sq) && isInput(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) =
+ rewriteExistentialExprWithAttrs(Seq(sq), p, rejectsUnknownResult = rejectsUnknownResult)
+ newCondition = newCondition.transform {
+ case e if e.fastEquals(sq) => rewritten.get
+ }
+ newInput
+ }
+ }
+ val newLeft = route(join.left, referencesInput(_, join.left))
+ val newRight = route(join.right, referencesInput(_, join.right))
+ if (newLeft.fastEquals(join.left) && newRight.fastEquals(join.right)) {
+ join
+ } else {
+ // Drop the `exists` attributes the nested existence joins introduced, keeping this join's
+ // own output, which already carries its own.
+ Project(join.output, join.copy(
+ left = newLeft, right = newRight, condition = Some(newCondition)))
+ }
+ }
+
+ /**
+ * Collects the existential sub-queries carried by the condition of an existence join, as the
+ * units [[rewriteExistentialExprWithAttrs]] rewrites: a NOT IN is collected as the [[Not]] over
+ * the [[InSubquery]], so the rewrite takes its null-aware arm and the unknown-result collector,
+ * which reports the same node, can be compared against it.
+ *
+ * A collected sub-query is not descended into. One nested below it is reached when it is
+ * rewritten, since the join built for it goes through the routing in turn, and one below a
+ * sub-query left in place is left with it, as everywhere else in this rule.
+ */
+ private def collectCarriedExistentialExprs(
+ expr: Expression,
+ collected: ArrayBuffer[Expression]): Unit = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return
+ }
+ expr match {
+ case sq @ Not(_: InSubquery) => collected += sq
+ case sq: Exists => collected += sq
+ case sq: InSubquery => collected += sq
+ case other => other.children.foreach(collectCarriedExistentialExprs(_, collected))
+ }
+ }
+
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true,
+ rejectsUnknownResult: Boolean = false): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan = rewriteCarriedExistentialExprs(
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint),
+ rejectsUnknownResult)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint), rejectsUnknownResult)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(
+ Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint), rejectsUnknownResult)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. A nested sub-query resolves in the scope
+ * of the sub-query it is nested in, so an attribute both plans produce is one of that
+ * sub-query's own.
+ *
+ * The probes are exclusive, so swapping their order would attribute such an attribute to
+ * `outerPlan` alone: `subPlanOnly` and `bothPlans` would both be false, and a nested sub-query
+ * whose only reference is the shared attribute would be declined and left in the join condition
+ * rather than routed to the sub-query plan that can evaluate it.
+ *
+ * No plan the analyzer produces from SQL reaches this: `DeduplicateRelations` has given the
+ * sub-query its own ExprIds, and the EXISTS arms classify against the raw sub-query plan while
+ * the IN arms classify against the plan `dedupSubqueryOnSelfJoin` has already separated from the
+ * outer plan. What does reach it is the shared-attribute fixture in `RewriteSubquerySuite`,
+ * which builds the overlap directly because the analyzer is not involved there. That fixture
+ * pins the shape, not the outcome of this ordering: under either order it ends in the same
+ * error, since `dedupSubqueryOnSelfJoin`'s pre-existing conflict check reports the duplicated
+ * attribute that the hoisted condition mentions.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite here.
+ * That does not leave the sub-query unexamined: one carried by a nested sub-query that is
+ * rewritten is routed against the inputs of the join built for it, see
+ * [[rewriteCarriedExistentialExprs]].
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * The rejection covers the sub-queries the sub-query plan owns. An IN sub-query that only
+ * references the outer plan is still rewritten two-valued, as it was before SPARK-59351, which
+ * SPARK-59862 tracks.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan,
+ keepsThreeValuedResiduals: Boolean = false): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(sq)) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above. Unless the caller keeps such
+ // a sub-query as a residual, in which case no existence join is built for it and there is
+ // nothing to reject: see `keepsThreeValuedResiduals`.
+ val unknownSensitive = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(ownedBySubPlan)
+ val (residual, unknownResult) = unknownSensitive.partition(sq =>
Review Comment:
**Non-blocking (P2):** A truth-only, uncorrelated nested NOT IN with a nullable comparison is now rewritten here into a null-aware existence join on the sub-query plan, where the merge base left it in the anti-join condition. For example, in `SELECT * FROM t1 WHERE a NOT IN (SELECT c1 FROM t2 WHERE c1 > a OR c1 NOT IN (SELECT col1 FROM t3))`, the join condition `(c1 = col1) OR isnull(c1 = col1)` has no equi key, so it is planned as a BroadcastNestedLoopJoin over t2 x t3. At the merge base, InSubqueryExec answered the nested NOT IN three-valued with a hash-set probe. The answer is unchanged, but the query now does |t2| x |t3| extra condition evaluations, a cost that grows with the nested list and buys no correctness.
This applies whenever no later join-condition rewrite reaches the anti join: when this NOT IN is the last sub-query conjunct of its Filter, or when `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` or `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is off. Under either setting off, the carried router does the same to uncorrelated truth-only units on the pre-existing routes. Where a later sub-query conjunct wraps the anti join under the default configuration, the `case j: Join` arm would rewrite the IN under the NOT positively and make the nested NOT IN two-valued, so the null-aware rewrite is needed there.
This extends an [existing discussion](https://github.com/apache/spark/pull/58656#discussion_r4281437494).
**Recommended change:** Keep uncorrelated nested IN and NOT IN sub-queries where the merge base evaluated them correctly, and rewrite them only where the fix needs it. In the NOT IN arm, keep as a residual in the anti-join condition every uncorrelated, sub-query-plan-owned nested IN or NOT IN that carries no correlated sub-query, not only the ones whose unknown is observable, whenever no later join-condition rewrite reaches that anti join: when this NOT IN is the last sub-query conjunct of its Filter, or when spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled or spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled is false. Otherwise keep today's handling, whose null-aware rewrite of a truth-only nested NOT IN is what keeps the join-condition rewrite from making it two-valued. In the carried router, on routes that do not reject an unknown result, leave an uncorrelated carried IN or NOT IN in place whenever the join-condition handling w
ould not rewrite it, that is when either of those settings is false. Update the NOT IN residual test so its truth-only case asserts the residual instead of an existence join, and add plan assertions for a nested NOT IN with a nullable comparison in the predicate-subquery tests.
**Why this works:** InSubqueryExec already evaluates an uncorrelated sub-query that stays in a bare anti-join or existence-join condition three-valued, with a hash-set probe, and needs no existence join. Leaving such sub-queries in place wherever no later join-condition rewrite reaches them reproduces the merge-base plan exactly where the merge base was already correct. Where that rewrite does reach them, the existing null-aware rewrite stays. Correlated sub-queries, sub-queries carrying a correlated descendant, and every sub-query on the sub-query-plan route still go through the existing routing and rejections.
**Scope:** Residual and in-place policy for uncorrelated nested IN and NOT IN in RewritePredicateSubquery, with updated NOT IN residual and carried-route tests.
**Compatibility:** Every result this head already gets right stays the same. The sub-queries this repair leaves in place are evaluated three-valued by InSubqueryExec, as at the merge base, and only where no later join-condition rewrite reaches them; every other one keeps today's rewrite. Only plans differ, and only where the conjunct position or the join-condition settings already decide whether uncorrelated IN sub-queries in join conditions are rewritten.
**Risks:** Plans for these shapes go back to InSubqueryExec, which collects and broadcasts the nested list on the driver whatever its size, as at the merge base. The change can be rolled back by reverting the residual policy alone. Reading the join-condition settings in the carried router duplicates the gating of the join-condition arm, so the two must stay aligned if that gating changes. Keying the NOT IN arm's residuals on its position in the conjunct fold ties the arm to that fold order, so the two must stay aligned if the fold changes.
**Constraints:** Correlated nested sub-queries, and those carrying a correlated descendant, must still be routed or rejected, because PlanSubqueries drops their hoisted condition. The EXISTS, NOT EXISTS and IN arms and the sub-query-plan route keep their current routing and unknown-result rejection. An Exists is never kept as a residual. Under the default configuration, a truth-only uncorrelated nested NOT IN under an outer NOT IN that a later sub-query conjunct wraps keeps its null-aware rewrite.
**Exclusions:**
- An unknown-observing residual under an outer NOT IN that a later sub-query conjunct wraps is still made two-valued by the join-condition rewrite under the default configuration, as at the merge base; that is left to SPARK-59862.
**Success:** An uncorrelated, truth-only, sub-query-plan-owned nested IN or NOT IN under an outer NOT IN that is the last sub-query conjunct is evaluated in the anti-join condition through InSubqueryExec, and no existence join is built for it. Under the default configuration, a truth-only uncorrelated nested NOT IN under an outer NOT IN followed by another sub-query conjunct is still rewritten null-aware and returns the standard SQL answer. With either join-condition setting disabled, an uncorrelated carried IN or NOT IN on the Filter-predicate, projection, join-condition or outer route stays in its existence-join condition. Every predicate-subquery test answer is unchanged.
**Verification:**
- **Regression:** For SELECT * FROM t1 WHERE a NOT IN (SELECT c1 FROM t2 WHERE c1 > a OR c1 NOT IN (SELECT col1 FROM t3)) with a nullable t3.col1, the executed plan contains no nested-loop existence join and plans the nested sub-query as an in-subquery filter, and the answer matches standard SQL.
- **Regression:** For a truth-only uncorrelated nested positive IN under an outer NOT IN, no existence join is built on the sub-query side and the answer is unchanged.
- **Behavior:** With decorrelatePredicateSubqueriesInJoinPredicate.enabled=false, and separately with optimizeUncorrelatedInSubqueriesInJoinCondition.enabled=false, an uncorrelated carried NOT IN with a nullable comparison on the Filter-predicate route stays in its existence-join condition, and the query returns the standard SQL answer.
- **Behavior:** Under the default configuration, SELECT * FROM t1 WHERE a NOT IN (SELECT c1 FROM t2 WHERE c1 > a OR c1 NOT IN (SELECT col1 FROM t3n)) AND EXISTS (SELECT 1 FROM t2 WHERE c1 >= a), with a NULL in t3n that matches nothing, returns the standard SQL answer, as it does on this head.
- **Compatibility:** Correlated nested sub-queries under an outer NOT IN are still rewritten or rejected as today. The carried NOT IN, shared bare-and-NOT IN and carried routing cases keep their answers under both values of each join-condition setting.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -184,8 +188,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Use EXISTS if performance matters to you.
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ // The hoisted correlated predicates may carry existential sub-queries that only the
+ // sub-query plan can evaluate, see rewriteExistentialExprInJoinCondition.
+ // This arm returns a bare Join, which this rule does not offer to its own handling of
+ // predicate sub-queries in join conditions, so an uncorrelated sub-query left in the
+ // anti-join condition stays there and `InSubqueryExec` evaluates it with the
+ // three-valued semantics an existence join cannot carry. That is what this arm did
+ // before SPARK-59351, so it is not rejected here. It only holds while this NOT IN is
+ // the last sub-query conjunct of the Filter: the fold over the conjuncts wraps this
+ // anti join as `p` of a later EXISTS or IN arm, in `Project(p.output, Join(p, ...))`,
Review Comment:
**Nit (P3):** The last-conjunct condition is right, but any later sub-query conjunct wraps this anti join, not only an EXISTS or IN. A later NOT IN arm builds a bare `Join` with this anti join as its left input, and the generic `case (p, predicate)` arm builds `Project(p.output, Filter(..., existence joins over p))`. In both cases the top-down pass then reaches the anti join in the `case j: Join` arm. For example, in `a NOT IN (...) AND b NOT IN (...)`, the first NOT IN's residual is rewritten two-valued under the default configuration. Saying 'a later sub-query conjunct' here, and in the comment of the 'NOT IN keeps an uncorrelated three-valued residual' test, would cover these cases.
This extends an [existing discussion](https://github.com/apache/spark/pull/58656#discussion_r4346076714).
**Verification:**
- **Inspection:** The comments' explanation covers a later NOT IN conjunct and a generic predicate conjunct as well as EXISTS and IN, matching how the fold builds each arm's result.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +417,649 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, a correlated IN or NOT IN is planned by `PlanSubqueries`
+ * as an in-subquery filter that drops its own hoisted condition, which returns extra rows, and
+ * a correlated EXISTS reaches execution as an unevaluable `Exists` and fails.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ *
+ * The unknown-result rejection applies only when `rejectsUnknownResult` is set, which is the
+ * case for the existence joins built on the sub-query-plan route, where the same shapes failed
+ * with an internal error. On the routes that predate this change an uncorrelated IN whose
+ * unknown result is observable is left where it is, as it was, and a correlated one is routed
+ * without the rejection. Their two-valued handling is tracked by SPARK-59862.
+ *
+ * A NOT IN is routed before a bare IN, since the substitution of a unit replaces every
+ * occurrence of it, including one inside a `Not` over the same instance.
+ */
+ private def rewriteCarriedExistentialExprs(
+ join: Join,
+ rejectsUnknownResult: Boolean): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
Review Comment:
**Nit (P3):** `referencesInput` rebuilds `effectiveReferences(sq)` and probes one input on every call. Through `referencesBothInputs`, `referencesOneInput` and the two `route` filters, that happens about ten times per carried unit, although `join.left` and `join.right` are fixed for the whole call. Each rebuild is proportional to the unit's reference count, so a wide row-valued IN pays for it repeatedly. `rewriteExistentialExprInSubqueryPlan` already classifies each candidate once and memoizes the result. Computing each unit's effective references once here, and deriving the left, right, one-input and both-inputs answers from that result, would match it.
This extends an [existing discussion](https://github.com/apache/spark/pull/58656#discussion_r4248986929).
**Verification:**
- **Inspection:** Every left or right input classification of a carried unit reads one per-unit result rather than rebuilding the effective-reference set.
- **Compatibility:** Routing, both rejections and the in-place filter give the same results for the carried routing, carried NOT IN, shared IN and both-inputs cases.
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,822 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. The EXISTS form of this query fails on the same condition, raised by the same
+ // rejection in rewriteExistentialExprInSubqueryPlan before any join exists; were that
+ // rejection removed, the EXISTS form would instead reach the join-condition handling, which
+ // rejects it with UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION under the default
+ // configuration, so only the NOT IN form pins this rejection on its own.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: existential subquery carried by a rewritten nested subquery") {
+ // The nested IN over t4 is owned by the subquery plan, so it is rewritten into an existence
+ // join and the condition PullupCorrelatedPredicates hoisted into it is grafted onto that
+ // join. That condition carries a third-level IN correlated to t4, which references neither
+ // the outer nor the subquery plan and so is routed by neither of the guards above. It has to
+ // be rewritten against the inputs of the join just built: left in place, PlanSubqueries plans
+ // it as an in-subquery filter and drops its own correlated predicate col1 = k, which returns
+ // every row of t1 instead of the one row where the correlation holds.
+ withTempView("t1", "t2", "t4", "t5") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ // No k equals any c1, so only the third-level IN can satisfy the hoisted disjunction, and
+ // its correlated predicate col1 = k is false for every row of t4: dropping it makes the
+ // level-2 IN hold for c1 = 1 and returns 1, 2 and 3.
+ Seq((1, 100), (9, 7)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(1).toDF("col1").persist().createOrReplaceTempView("t5")
+ val query =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t4
+ | WHERE k = c1 OR col1 IN (SELECT col1 FROM t5 WHERE col1 = k)))""".stripMargin
+ // The routing must not depend on the rewrite of predicate subqueries in join conditions,
+ // which is what used to make these two configurations disagree: it reached the grafted
+ // condition only when enabled.
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswerAndPlan(query, Row(1) :: Nil)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: a carried subquery is classified as on the subquery-plan route") {
+ // The carried router applies the same rules as rewriteExistentialExprInSubqueryPlan, against
+ // the inputs of the existence join rather than the outer and the subquery plan.
+ withTempView("t1", "t2", "t4", "t5", "t3n", "t5n") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((9, 7)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(Some(3), None).toDF("x").persist().createOrReplaceTempView("t3n")
+ Seq((Some(3), 7)).toDF("x", "k").persist().createOrReplaceTempView("t5n")
+
+ // A carried NOT IN is collected as the Not over the IN, so the rewrite takes its null-aware
+ // arm. Taking the positive-IN arm instead leaves NOT exists over a condition that ignores
+ // NULL comparisons: 5 NOT IN (NULL) is unknown, so n4's (5, 100) row is excluded and the
+ // answer is empty, where the two-valued form returns every row of n1, that is 1 and 2.
+ withTempView("n1", "n2", "n4", "n5") {
+ Seq(1, 2).toDF("a").persist().createOrReplaceTempView("n1")
+ Seq(5).toDF("c1").persist().createOrReplaceTempView("n2")
+ Seq((5, 100)).toDF("col1", "k").persist().createOrReplaceTempView("n4")
+ Seq((Option.empty[Int], 100)).toDF("x", "y").persist().createOrReplaceTempView("n5")
+ val carriedNotIn =
+ """SELECT * FROM n1 WHERE EXISTS (
+ | SELECT 1 FROM n2 WHERE a = c1 OR c1 IN (
+ | SELECT col1 FROM n4
+ | WHERE k = c1 OR col1 NOT IN (SELECT x FROM n5 WHERE y = k)))""".stripMargin
+ Seq("true", "false").foreach { conf =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> conf) {
+ checkAnswer(sql(carriedNotIn), Nil)
+ }
+ }
+ }
+
+ // An uncorrelated carried IN is routed too, not left for the join-condition rewrite, which
+ // has no unknown guard and would make it a two-valued exists under the default
+ // configuration. Its unknown is observable here, so it is rejected under both settings.
+ val uncorrelatedUnknown =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (
+ | SELECT col1 FROM t4
+ | WHERE k = c1 OR ((col1 IN (SELECT x FROM t3n)) <=> false)))""".stripMargin
+ // A correlated one in the same position, which must report the same condition rather than
+ // the one about referencing both queries.
+ val correlatedUnknown =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (
+ | SELECT col1 FROM t4
+ | WHERE k = c1
+ | OR ((col1 IN (SELECT x FROM t5n WHERE x = k)) <=> false)))""".stripMargin
+ Seq(uncorrelatedUnknown, correlatedUnknown).foreach { query =>
+ Seq("true", "false").foreach { conf =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> conf) {
+ val e = intercept[AnalysisException](sql(query).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT", query)
+ }
+ }
+ }
+
+ // The same shape where only truth is observed is still rewritten and answers correctly.
+ Seq("true", "false").foreach { conf =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> conf) {
+ checkAnswer(
+ sql("""SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (
+ | SELECT col1 FROM t4
+ | WHERE k = c1 OR (col1 IN (SELECT x FROM t3n))))""".stripMargin),
+ Row(1) :: Nil)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: carried subqueries are routed on every route that builds an existence join") {
+ // rewriteCarriedExistentialExprs runs on every existence join this rule builds, not only on
+ // the ones the semi/anti join arms build for a top-level predicate subquery. On the routes
+ // below the outer EXISTS is itself rewritten into an existence join by
+ // rewriteExistentialExprWithAttrs, so the nested IN hoisted into its condition is carried
+ // rather than routed by the first-level classification, and only this router reaches it.
+ withTempView("t1", "t2", "t4", "t6") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ // The correlated nested IN is false for every c1 of t2, while the same IN without k = c1
+ // holds for c1 = 1, so losing that predicate turns every answer below into every row.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(1, 2, 3).toDF("b").persist().createOrReplaceTempView("t6")
+ val exists =
+ "EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t4 WHERE k = c1))"
+
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ // The Filter-predicate arm: under OR the EXISTS is not a conjunct of its own, so it
+ // goes through rewriteExistentialExpr rather than through the semi join arm.
+ checkAnswer(sql(s"SELECT * FROM t1 WHERE a > 100 OR $exists"), Row(1) :: Nil)
+ // handleUnaryNode: the EXISTS is a projection rather than a predicate.
+ checkAnswer(
+ sql(s"SELECT a, $exists AS e FROM t1"),
+ Row(1, true) :: Row(2, false) :: Row(3, false) :: Nil)
+ }
+ }
+
+ // The join-condition arm reaches the router only when it runs at all, which is what
+ // decorrelatePredicateSubqueriesInJoinPredicate gates; with it disabled the subquery stays
+ // in the join condition, which is pre-existing behaviour this PR does not change.
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "true") {
+ checkAnswer(
+ sql(s"SELECT t1.a FROM t1 JOIN t6 ON t1.a = t6.b AND $exists"), Row(1) :: Nil)
+ }
+ }
+ }
+
+ test("SPARK-59351: a carried IN shared between a bare and a negated position") {
+ // Filter pushdown substitutes the alias, so the same IN instance is carried bare and under
+ // a Not. Routing the bare one must not replace the one inside the Not, or the null-aware
+ // existence join built for the Not is left unreferenced and NOT exists over a two-valued
+ // match returns the rows where the IN is unknown. Standard SQL keeps only a = 1.
+ withTempView("s1", "s2", "s3") {
+ Seq(Some(1), Some(-1), None).toDF("a").persist().createOrReplaceTempView("s1")
+ Seq(1).toDF("c1").persist().createOrReplaceTempView("s2")
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("s3")
+ val query =
+ """SELECT s.a FROM (SELECT a, a IN (SELECT col1 FROM s3) AS b FROM s1) s
+ |WHERE s.a > 0 OR EXISTS (SELECT 1 FROM s2 WHERE s.a = s2.c1 OR s.b OR NOT s.b)"""
+ .stripMargin
+ Seq("true", "false").foreach { conf =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> conf) {
+ checkAnswer(sql(query), Row(1) :: Nil)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: a carried subquery referencing both inputs of its existence join") {
+ // The third-level IN takes its compared value c1 from the left input of the existence join
+ // built for the nested IN over t4, and its hoisted x = k reads the right one. Neither input
+ // can evaluate it, and left in place it is planned as an in-subquery filter that drops x = k.
+ withTempView("t1", "t2", "t4", "t5") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(1, 9).toDF("x").persist().createOrReplaceTempView("t5")
+ val query =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (
+ | SELECT col1 FROM t4
+ | WHERE k = c1 OR c1 IN (SELECT x FROM t5 WHERE x = k)))""".stripMargin
+ Seq("true", "false").foreach { conf =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> conf) {
+ val e = intercept[AnalysisException](sql(query).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY", conf)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: carried unknown-result rejection is limited to the subquery-plan route") {
+ // On the routes that predate this change the carried IN below was answered, and it is not
+ // rejected: its handling there is left to SPARK-59862. The columns are nullable but hold no
+ // NULL, so the two-valued and the three-valued answers agree and a rejection would be the
+ // only difference. The first query takes its compared value from the right input of the
+ // existence join built for the EXISTS, the second from the left one.
+ withTempView("t1", "t2", "t3") {
+ Seq(Some(1), Some(2), Some(3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(Some(1), Some(8), Some(9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq(Some(8)).toDF("col1").persist().createOrReplaceTempView("t3")
Review Comment:
**Nit (P3):** With `t3` holding only 8, the two-valued and three-valued answers agree, as the test comment says, so this case shows only that nothing is rejected. Removing the `filterNot` that leaves these INs in place would leave every assertion passing. Adding a non-matching NULL to `t3` and asserting under `decorrelatePredicateSubqueriesInJoinPredicate.enabled=false` would pin it. There the rightInput query returns only 1 when the IN stays in place, and 1, 2 and 3 after a two-valued rewrite. Under the default setting both forms return 1, 2 and 3 (SPARK-59862), so only the disabled setting can tell them apart.
**Verification:**
- **Regression:** With a list column holding a non-matching NULL and the join-condition setting disabled, the right-input and left-input queries on the Filter-predicate route return the three-valued answer. The case fails if the uncorrelated unknown-observing carried IN is routed into a two-valued existence join.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,568 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is deliberately left where it is. It has no hoisted condition to
+ * lose, and `InSubqueryExec` evaluates it with the three-valued semantics that the `exists`
+ * attribute of an existence join cannot carry.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ // Only a correlated sub-query has to be consumed here, see above.
+ val carried = condition.collect {
+ case sq @ (_: Exists | _: InSubquery) if hasCorrelatedCondition(sq) => sq
+ }
+ if (carried.isEmpty) {
+ return join
+ }
+ // Where the value of a carried IN is observed beyond being TRUE, an existence join cannot
+ // stand in for it, exactly as on the routes above.
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+
+ val rejected = carried.filter { sq =>
+ observesUnknown(sq) ||
+ (referencesInput(sq, join.left) && referencesInput(sq, join.right))
+ }
+ if (rejected.nonEmpty) {
+ // A sub-query that needs both inputs can be rewritten against neither, and one whose
+ // unknown is observable cannot become an `exists` attribute at all.
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(rejected)
+ }
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ carried.filter(sq => isInput(sq) && !rejected.contains(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) = rewriteExistentialExprWithAttrs(Seq(sq), p)
Review Comment:
Confirmed on 2666748431d: collectCarriedExistentialExprs collects a NOT IN as the Not over its IN, so the router takes the null-aware arm. observesUnknown now compares against the same Not node the unknown collector reports.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4346076647","thread_id":"inline:4346076647","verdict_sha256":"596d6ef5f193363401f54af770e62e1e49b3bfd27c84efadb74d430a0c108603"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,568 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is deliberately left where it is. It has no hoisted condition to
+ * lose, and `InSubqueryExec` evaluates it with the three-valued semantics that the `exists`
+ * attribute of an existence join cannot carry.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ // Only a correlated sub-query has to be consumed here, see above.
+ val carried = condition.collect {
+ case sq @ (_: Exists | _: InSubquery) if hasCorrelatedCondition(sq) => sq
Review Comment:
Confirmed on 2666748431d: the router now considers uncorrelated carried INs, and on the sub-query-plan route this shape is rejected with NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT under both settings. The later narrowing of that rejection to the sub-query-plan route leaves the older routes to SPARK-59862, as discussed.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4346076676","thread_id":"inline:4346076676","verdict_sha256":"596d6ef5f193363401f54af770e62e1e49b3bfd27c84efadb74d430a0c108603"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,568 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is deliberately left where it is. It has no hoisted condition to
+ * lose, and `InSubqueryExec` evaluates it with the three-valued semantics that the `exists`
+ * attribute of an existence join cannot carry.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ // Only a correlated sub-query has to be consumed here, see above.
+ val carried = condition.collect {
+ case sq @ (_: Exists | _: InSubquery) if hasCorrelatedCondition(sq) => sq
+ }
+ if (carried.isEmpty) {
+ return join
+ }
+ // Where the value of a carried IN is observed beyond being TRUE, an existence join cannot
+ // stand in for it, exactly as on the routes above.
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+
+ val rejected = carried.filter { sq =>
+ observesUnknown(sq) ||
+ (referencesInput(sq, join.left) && referencesInput(sq, join.right))
+ }
+ if (rejected.nonEmpty) {
+ // A sub-query that needs both inputs can be rewritten against neither, and one whose
+ // unknown is observable cannot become an `exists` attribute at all.
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(rejected)
Review Comment:
Confirmed: the carried router raises NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY only for a sub-query that references both inputs, and NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT for the unknown-result case. The new tests reach both.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4346076695","thread_id":"inline:4346076695","verdict_sha256":"596d6ef5f193363401f54af770e62e1e49b3bfd27c84efadb74d430a0c108603"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4380505336
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,749 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. The EXISTS form of this query fails on the same condition, raised by the same
+ // rejection in rewriteExistentialExprInSubqueryPlan before any join exists; were that
+ // rejection removed, the EXISTS form would instead reach the join-condition handling, which
+ // rejects it with UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION under the default
+ // configuration, so only the NOT IN form pins this rejection on its own.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: existential subquery carried by a rewritten nested subquery") {
+ // The nested IN over t4 is owned by the subquery plan, so it is rewritten into an existence
+ // join and the condition PullupCorrelatedPredicates hoisted into it is grafted onto that
+ // join. That condition carries a third-level IN correlated to t4, which references neither
+ // the outer nor the subquery plan and so is routed by neither of the guards above. It has to
+ // be rewritten against the inputs of the join just built: left in place, PlanSubqueries plans
+ // it as an in-subquery filter and drops its own correlated predicate col1 = k, which returns
+ // every row of t1 instead of the one row where the correlation holds.
+ withTempView("t1", "t2", "t4", "t5") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ // No k equals any c1, so only the third-level IN can satisfy the hoisted disjunction, and
+ // its correlated predicate col1 = k is false for every row of t4: dropping it makes the
+ // level-2 IN hold for c1 = 1 and returns 1, 2 and 3.
+ Seq((1, 100), (9, 7)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(1).toDF("col1").persist().createOrReplaceTempView("t5")
+ val query =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t4
+ | WHERE k = c1 OR col1 IN (SELECT col1 FROM t5 WHERE col1 = k)))""".stripMargin
+ // The routing must not depend on the rewrite of predicate subqueries in join conditions,
+ // which is what used to make these two configurations disagree: it reached the grafted
+ // condition only when enabled.
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswerAndPlan(query, Row(1) :: Nil)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: a carried subquery is classified as on the subquery-plan route") {
+ // The carried router applies the same rules as rewriteExistentialExprInSubqueryPlan, against
+ // the inputs of the existence join rather than the outer and the subquery plan.
+ withTempView("t1", "t2", "t4", "t5", "t3n", "t5n") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((9, 7)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(Some(3), None).toDF("x").persist().createOrReplaceTempView("t3n")
+ Seq((Some(3), 7)).toDF("x", "k").persist().createOrReplaceTempView("t5n")
+
+ // A carried NOT IN is collected as the Not over the IN, so the rewrite takes its null-aware
+ // arm. Taking the positive-IN arm instead leaves NOT exists over a condition that ignores
+ // NULL comparisons: 5 NOT IN (NULL) is unknown, so the t4 row is excluded and the answer is
Review Comment:
Fixed. The comment now names n4's `(5, 100)` as the excluded row and n1's 1 and 2 as the "every row" answer of the two-valued form, so it can be checked against the fixtures defined just below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4056573515
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,100 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, so it is left in the join condition, where both plans are in scope. This is only
+ * correct for an uncorrelated sub-query: the join condition of a correlated one is dropped when
+ * it is planned as an in-subquery filter, which would silently change the result, so it is
+ * reported as unsupported instead. Note that such a sub-query can be correlated only to the
+ * sub-query plan, as being correlated to the outer plan as well would require two levels of
+ * correlation, which the Analyzer rejects.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A correlated sub-query referencing both plans has to stay in the join condition, where its
+ // own join condition is lost, so reject it rather than silently return a wrong result.
+ val unsupported = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) &&
+ referencesPlan(sq, subPlan) && referencesPlan(sq, outerPlan) => sq
+ })
+ if (unsupported.nonEmpty) {
+ throw QueryCompilationErrors.unsupportedCorrelatedSubqueryInJoinConditionError(unsupported)
Review Comment:
Agreed — "join predicate" and "join inputs" name nothing in a query whose SQL contains no join.
Rather than reword the existing sub-condition, which is right for the site at `subquery.scala:249`
where the user did write a JOIN, this adds two sibling sub-conditions under the same `0A000`
category and uses them at the new sites:
* `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY` — "The subquery `<subqueryExpression>` is
nested in a correlated condition of another subquery, and references both the outer query and the
subquery it is nested in."
* `NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT` — "The IN subquery `<subqueryExpression>` is nested in a
correlated condition of another subquery, and its result can be unknown where it is not used as a
predicate."
The existing sub-condition keeps its JOIN wording and its call site. Names and wording are of course
open to your preference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4057052748
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
Review Comment:
**Nit (P3):** The code-formatted configuration name is missing its `.enabled` suffix. The registered key is `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`; please use that exact name here.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) && referencesPlan(sq, subPlan) &&
Review Comment:
**Non-blocking (P2):** `listQuery.isCorrelated` is based on retained `outerAttrs`, not the surviving pulled-up condition. `PullupCorrelatedPredicates` deliberately keeps wrapper state for idempotency, so a query such as `a = c1 OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)` reaches this guard with no effective nested correlation but still has `outerAttrs = c1`; this branch then rejects the valid, effectively uncorrelated IN as referencing both plans. Please classify correlation from the surviving condition and add this simplification case to the regression tests, while preserving rejection of the genuinely correlated shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4046157769
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3227,198 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n") {
+ Seq((1), (2), (3), (7)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+
+ // Checks the result, and that every node of the optimized plan can produce the attributes
+ // it references. The latter is what this fix is about: an existence join whose condition
+ // references an attribute produced by neither of its children can still return the right
+ // answer when the nested relation is empty at runtime, because the invalid condition is
+ // then never bound. checkAnswer alone would not catch it, as the missing input checks it
+ // runs only look at the root of the plan.
+ def checkAnswerAndPlan(query: String, expected: Seq[Row]): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ checkAnswer(df, expected)
+ }
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // Same, with a nested subquery that returns no matching row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(1) :: Nil)
Review Comment:
**MEDIUM**
query4, query5 and query8 hoist `a = c1`, the key equality itself, so their answers cannot depend on the nested subquery: `a IN {c1 ∈ t2 : a = c1 OR nested(c1)}` reduces to `a ∈ t2` whatever `nested` returns. query8 is worse: `c1 IN (SELECT col1 FROM t3 WHERE col1 = c1)` is logically identical to the uncorrelated `c1 IN (SELECT col1 FROM t3)`, so no data distinguishes correlated nesting from uncorrelated, the property the description lists it as covering.
query4 and query8 still carry the `missingInput` half; query5 does not, since `SubqueryExpression.references` excludes the join condition, so the nested `Exists` there has `outerAttrs = {c1}`, produced by the right child. Its pre-fix failure is an `Unevaluable` `Exists` reaching evaluation. So the NOT IN branch rests on "it no longer throws" alone.
Making the answers depend on the nested subquery needs the hoisted disjunct to stop being the key equality, plus new data: `a > c1 OR ...` reduces query4 to `a ∈ t2 AND a ∈ t3`, empty on these rows. A second column on `t3`, correlated on the one it does not project, removes query8's degenerate correlation, though its answer stays insensitive either way.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,100 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, so it is left in the join condition, where both plans are in scope. This is only
+ * correct for an uncorrelated sub-query: the join condition of a correlated one is dropped when
+ * it is planned as an in-subquery filter, which would silently change the result, so it is
+ * reported as unsupported instead. Note that such a sub-query can be correlated only to the
+ * sub-query plan, as being correlated to the outer plan as well would require two levels of
+ * correlation, which the Analyzer rejects.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A correlated sub-query referencing both plans has to stay in the join condition, where its
+ // own join condition is lost, so reject it rather than silently return a wrong result.
+ val unsupported = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) &&
+ referencesPlan(sq, subPlan) && referencesPlan(sq, outerPlan) => sq
+ })
+ if (unsupported.nonEmpty) {
+ throw QueryCompilationErrors.unsupportedCorrelatedSubqueryInJoinConditionError(unsupported)
Review Comment:
**LOW**
`UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION` reads "Correlated subqueries in the join predicate cannot reference both join inputs" (`error/error-conditions.json:9756`). That reads as written for the call site at `subquery.scala:249`, where the user wrote a JOIN, so "join predicate" and "join inputs" name things in their query. At this call site the join is the semi/anti join Spark synthesizes from `EXISTS` or `IN`, and the user's SQL contains no join at all.
Both call sites raise it off a join Spark synthesized: this one, and the subquery that stays in the join condition under the default configuration and is rejected at `:249`. The category is right and only the sub-condition wording does not transfer. Could it say the subquery cannot reference both the outer query and the subquery it is nested in, or gain a sibling sub-condition? Either one covers both sites.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,100 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, so it is left in the join condition, where both plans are in scope. This is only
+ * correct for an uncorrelated sub-query: the join condition of a correlated one is dropped when
+ * it is planned as an in-subquery filter, which would silently change the result, so it is
+ * reported as unsupported instead. Note that such a sub-query can be correlated only to the
+ * sub-query plan, as being correlated to the outer plan as well would require two levels of
+ * correlation, which the Analyzer rejects.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A correlated sub-query referencing both plans has to stay in the join condition, where its
+ // own join condition is lost, so reject it rather than silently return a wrong result.
+ val unsupported = conditions.flatMap(_.collect {
Review Comment:
**LOW**
This `collect` walks the whole expression, including the join condition of a subquery that `canRewrite` declines, since `Exists.children` is `outerAttrs ++ joinCond`. The rewrite deliberately does not descend there, which is what the new scaladoc is about, so the rejection covers more ground than the rewrite does: a correlated subquery buried in a declined subquery's own join condition would fail the query even though nothing would have rewritten it.
Reaching it needs that inner subquery to reference both of the current plans, i.e. two levels of correlation, which the scaladoc says the Analyzer rejects, so I could not build a query that gets there: a scope mismatch rather than a live bug. One sentence here saying the wider scan is deliberate settles it. Narrowing the scan to match the rewrite would be the worse direction, since a subquery hidden under a declined one would then be planned without its join condition or reach execution unevaluable.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,100 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, so it is left in the join condition, where both plans are in scope. This is only
+ * correct for an uncorrelated sub-query: the join condition of a correlated one is dropped when
+ * it is planned as an in-subquery filter, which would silently change the result, so it is
+ * reported as unsupported instead. Note that such a sub-query can be correlated only to the
+ * sub-query plan, as being correlated to the outer plan as well would require two levels of
+ * correlation, which the Analyzer rejects.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
Review Comment:
**LOW**
This paragraph says a subquery referencing both plans "stays in the join condition, where both plans are in scope", and it does say that holds only for the uncorrelated case. What it leaves out is that the uncorrelated case then depends on `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`. The PR description has the rest: under the default configuration the pre-existing rewrite of predicate subqueries in join conditions rejects it with the same error, and it survives and returns the right answer only with that conf off. The comment does not carry that, and the test that pins the shape (`SubquerySuite.scala:3412`) sets the non-default value without saying why it has to.
The description sinks out of view once this merges, so the comment and the test are what the next reader has. One clause in the scaladoc naming the default-configuration rejection, and a line in the test saying the conf is there to bypass that branch, would keep the two in step.
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3227,198 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n") {
+ Seq((1), (2), (3), (7)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+
+ // Checks the result, and that every node of the optimized plan can produce the attributes
+ // it references. The latter is what this fix is about: an existence join whose condition
+ // references an attribute produced by neither of its children can still return the right
+ // answer when the nested relation is empty at runtime, because the invalid condition is
+ // then never bound. checkAnswer alone would not catch it, as the missing input checks it
+ // runs only look at the root of the plan.
+ def checkAnswerAndPlan(query: String, expected: Seq[Row]): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ checkAnswer(df, expected)
+ }
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // Same, with a nested subquery that returns no matching row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(1) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR EXISTS (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is
+ // never true, so only the correlated predicate can be satisfied.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Nil)
+
+ // Without the NULL, c1 NOT IN (3, 9) holds for c1 = 1.
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested subquery that is itself correlated to the query it is nested in.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(1) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // would drop its correlated predicate col1 = c1 and return an extra row, so it is rejected.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ // The rejection must not depend on whether predicate subqueries in join conditions are
+ // decorrelated, as that rewrite rejects the same shape on its own.
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
Review Comment:
**LOW**
Deleting the new `throw` at `subquery.scala:551` leaves the `true` iteration of this loop green: the nested subquery stays in the join condition, and since `RewritePredicateSubquery.apply` descends into the children of the node it just produced and the EXISTS arm returns `Project(p.output, join)`, the pre-existing branch at `subquery.scala:249` sees that join and raises the same condition.
The `false` iteration is the discriminating one: with `case j: Join` no longer matching, removing the new throw drops `col1 = c1`, an extra row comes back, and `intercept` gets no exception.
No exception assertion separates the two sites: both call the same method with one `subqueryExpression` parameter, and for this query both pass the same expression, so the rendered `.sql` is identical. While both sites raise the same condition the `true` iteration cannot be made to discriminate, so either the comment should say that it reaches `subquery.scala:249` and only the `false` iteration covers the new throw, or the case should use NOT IN as the outer predicate: that arm returns a bare `Join` (`subquery.scala:210`), which `transformDown` never re-offers to the `case j: Join` branch, so the new throw is the only one that can fire under either conf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5729170954
also cc @cloud-fan and @peter-toth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114363679
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,206 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) && !effectivelyReferencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
Review Comment:
Correct, and this was more than wording: I propagated the flag through `Not`. The contract is
narrowed to the positions where only truth is observed, `Not` no longer carries it, and the scaladoc
now states why a `NOT` directly over an IN is nonetheless safe — its null-aware join condition is
TRUE where the comparison is unknown, so the negated `exists` is FALSE there. Tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4274304907
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,476 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. The two plans share no ExprId in any
+ * shape reached by the suite: `DeduplicateRelations` has given the sub-query its own, and the
+ * EXISTS arms classify against the raw sub-query plan while the IN arms classify against the
+ * plan `dedupSubqueryOnSelfJoin` has already separated from the outer plan. The order is
+ * therefore defensive rather than something a query exercises, and it is the safe one: a
+ * nested sub-query resolves in the scope of the sub-query it is nested in, so an attribute
+ * both plans produce is one of that sub-query's own, and attributing it to `outerPlan` instead
+ * would let a single shared attribute satisfy both sides of the classification and have a
+ * nested sub-query referencing only the inner query rejected as referencing both.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite
+ * without rejecting the sub-query, as a sub-query referencing neither plan can be left in the
+ * join condition unharmed.
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(sq)) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(ownedBySubPlan)
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) =
+ rewriteExistentialExprWithAttrs(conditions, subPlan, ownedBySubPlan)
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ // COALESCE(x, false) maps both unknown and FALSE to FALSE, so only truth is observed. A
+ // different default does not: COALESCE(x, true) is TRUE where x is unknown and FALSE where x
+ // is FALSE.
+ case Coalesce(child +: rest) if rest.headOption.contains(Literal.FalseLiteral) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) => collectOrDescend(not, in, truthOnly, reported)
+ case Not(child) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = false, reported)
+ case in: InSubquery => collectOrDescend(in, in, truthOnly, reported)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = true, reported))
+ sq.getOuterAttrs.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ case other =>
+ other.children.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ }
+
+ /**
+ * Appends `report` to `reported` if the IN sub-query `in` would lose an observable unknown, and
+ * otherwise descends into the values it compares and into its list query. The descent matters
+ * because [[InSubquery]] is a [[Predicate]] rather than a [[SubqueryExpression]], so an IN
+ * sub-query nested in the values or in the hoisted join condition of another one is only reached
+ * from here.
+ */
+ private def collectOrDescend(
+ report: Expression,
+ in: InSubquery,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = {
+ if (!truthOnly && inSubqueryMayBeUnknown(in)) {
+ reported += report
+ } else {
+ in.values.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ collectUnknownSensitiveInSubqueries(in.query, truthOnly, reported)
+ }
+ }
+
+ /**
+ * Returns true if the row comparison of the IN sub-query `in` can evaluate to unknown, that is
+ * if it can return NULL rather than only TRUE or FALSE. An existence join cannot represent that
+ * third value, see [[rewriteExistentialExprInJoinCondition]].
+ *
+ * This deliberately does not use `InSubquery.nullable`, which under
+ * `spark.sql.legacy.inSubqueryNullability` (SPARK-43413) reports only the nullability of the
+ * compared values and ignores that of the list query's columns. What is protected here is the
+ * three-valued result the rewrite would drop, which does not depend on that configuration, so
+ * the nullability is re-derived from both sides.
+ */
+ private def inSubqueryMayBeUnknown(in: InSubquery): Boolean = {
+ // Iterators, so a wide row-valued IN stops at the first nullable side rather than pairing up
+ // the whole arity first.
+ in.values.iterator.zip(in.query.childOutputs.iterator).exists {
Review Comment:
Confirmed — every UNKNOWN-sensitive regression was scalar, so the all-fields contract was unprotected. Added, in both directions you asked for, with the nullability isolated to the second pair so the first pair is non-nullable on both sides:
* **later list output nullable** — `(c1, c2) IN (SELECT col1, col2 FROM t6)`, where `t2r` is non-nullable and only `t6.col2` is;
* **later compared value nullable** — `(c1, c2n) IN (SELECT col1, col2 FROM t7)`, where `t7` is non-nullable and only `t2rn.c2n` is.
Both are rejected. And the fully non-nullable row-valued shape, `(c1, c2) IN (SELECT col1, col2 FROM t7)`, is still rewritten and plan-checked, so the guard cannot be satisfied by giving up on arity in the other direction.
Checked that they discriminate: replacing the `zip` with `zip(...).take(1)`, so only the first pair is inspected, leaves both new rejections un-fired and the test red, while the whole pre-existing suite stays green.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4274247959
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,476 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. The two plans share no ExprId in any
+ * shape reached by the suite: `DeduplicateRelations` has given the sub-query its own, and the
+ * EXISTS arms classify against the raw sub-query plan while the IN arms classify against the
+ * plan `dedupSubqueryOnSelfJoin` has already separated from the outer plan. The order is
+ * therefore defensive rather than something a query exercises, and it is the safe one: a
+ * nested sub-query resolves in the scope of the sub-query it is nested in, so an attribute
+ * both plans produce is one of that sub-query's own, and attributing it to `outerPlan` instead
+ * would let a single shared attribute satisfy both sides of the classification and have a
+ * nested sub-query referencing only the inner query rejected as referencing both.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite
+ * without rejecting the sub-query, as a sub-query referencing neither plan can be left in the
+ * join condition unharmed.
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
Review Comment:
The shape reproduces and returns extra rows exactly as you describe, and it is a different hole from `r4248986904` — that one was a *declined both-plan* parent, this one is a *rewritten subplan-owned* parent whose carried descendant the atomic replacement never visits. So I am not marking it a duplicate.
But it is **pre-existing**, and this PR does not change it. With `t1(a) = 1,2,3`, `t2(c1) = 1,8,9`, `t4(col1,k) = (1,100),(9,7)`, `t5(col1) = 1`:
```sql
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2
WHERE a = c1
OR c1 IN (SELECT col1 FROM t4
WHERE k = c1 OR col1 IN (SELECT col1 FROM t5 WHERE col1 = k)));
```
The level-2 IN is subplan-owned and rewritten; the level-3 IN is correlated to `t4`, references neither the outer nor the subquery plan, is declined, and survives in the `ExistenceJoin` condition as `col1#125 IN (list#136 [k#126 && (col1#134 = k#126)])`. With join-predicate decorrelation disabled, `PlanSubqueries` drops that `joinCond` and the query returns `1, 2, 3`. DuckDB returns `1`.
I then ran the same query with **only** `RewritePredicateSubquery` reverted to its pre-PR version (`f3859964`), rebuilt, same fixtures:
| | `decorrelate...InJoinCondition = false` | `= true` (default) |
|---|---|---|
| pre-PR (`f3859964`) | `1, 2, 3` | `1` |
| this PR | `1, 2, 3` | `1` |
The optimized plans are **byte-identical** between the two, down to the `exists#141` ExprId. As a control that the revert was effective, the round-9 rejection test fails against the reverted rule, so the build did pick it up.
So the new routing neither introduces this nor makes it newly reachable: it is the pre-existing `PlanSubqueries` behaviour of dropping a surviving `ListQuery`'s hoisted condition, the same family as SPARK-59862.
One thing I want your call on before implementing, because the obvious fix is wrong. Rejecting every declined correlated descendant would **regress the default configuration**, which returns the correct answer `1` for this query today — the pre-existing join-condition handler rewrites the carried subquery into a second existence join. A blanket rejection would turn a working query into an error. So the fix has to be either:
1. rewrite the carried descendant against the newly built join, which is what the default path already achieves and would make the two configurations agree; or
2. reject only when nothing downstream will handle it, which means making the validation configuration-dependent.
I would rather do (1), and in its own JIRA, since it is the pre-existing drop that is wrong and the fix belongs next to `PlanSubqueries` rather than in this routing change. If you want it in this PR, I will take (1) here — but I do not think (2) should be written either way, @cloud-fan Could you please provide your views on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4287080702
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,476 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. The two plans share no ExprId in any
Review Comment:
Have addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6336192874
Thanks for the update. I re-reviewed 4168a32841b: correlated carried descendants are now routed, but the second case from r4281444233 still returns every row of t1, because an uncorrelated value-position IN carried by the existence join on the sub-query plan is left for the join-condition rewrite, which turns it into a two-valued exists; details are in the comment on optimizer/subquery.scala.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"conversation:6332638268","thread_id":"conversation:6332638268","verdict_sha256":"3bb54f2a3a4b20553e9d3ca5caee9c4a8fc61921fc05f7c2363e10d0f4e5e7b0"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4022391019
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -2678,4 +2678,133 @@ class SubquerySuite extends SharedSparkSession
assert(exposedAttribute.exprId == outerReferenceAttribute.exprId)
}
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n") {
+ Seq((1), (2), (3), (7)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ val df1 = sql(query1)
+ // Every plan node must be able to produce the attributes it references.
+ val invalidNodes = df1.queryExecution.optimizedPlan.collect {
+ case p if p.missingInput.nonEmpty => p
+ }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |${df1.queryExecution.optimizedPlan}""".stripMargin)
+ checkAnswer(df1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // Same, with a nested subquery that returns no matching row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswer(sql(query2), Row(1) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswer(sql(query3), Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswer(sql(query4), Row(1) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR EXISTS (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswer(sql(query5), Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is
+ // never true, so only the correlated predicate can be satisfied.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswer(sql(query6), Row(1) :: Nil)
+
+ // Without the NULL, c1 NOT IN (3, 9) holds for c1 = 1.
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswer(sql(query7), Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested subquery that is itself correlated to the query it is nested in.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswer(sql(query8), Row(1) :: Nil)
Review Comment:
The tree-wide `optimizedPlan.collect { case p if p.missingInput.nonEmpty => p }` on query1 is the only assertion in the new test that can catch an invalid plan before execution; the other seven queries only call `checkAnswer`. `assertEmptyMissingInput` inside `checkAnswer` looks at the roots of the analyzed/optimized/executed plans only, and `validateNoDanglingReferences` uses `collectFirst`, which matches at the root and returns before it ever reaches the `ExistenceJoin` below.
Your description notes that a nested subquery correlated to the query it is nested in can still produce the right answer on an invalid plan when the nested relation is empty at runtime, and query5 and query8 are both that shape with only a `checkAnswer` on each. Pulling the tree-wide check into a small helper and running it on those two would give this class of regression something to trip on.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -454,6 +470,70 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
}
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, so it is left in the join condition, where both plans are in scope. Such a
+ * sub-query is necessarily uncorrelated, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects. It is then either planned as
+ * an in-subquery filter or reported as unsupported by the rewrite of predicate sub-queries in
+ * join conditions.
Review Comment:
The doc says such a leftover is necessarily uncorrelated. That does not hold for `InSubquery`: it has no `references` override, so its references are those of its children (`values :+ query`), and the outer reference can come from `values` while the subquery stays correlated to the inner query.
`PlanSubqueries` ignores `joinCond` when it matches the list query, so the hoisted `col1 = c1` is gone. With `decorrelatePredicateSubqueriesInJoinPredicate` off, `EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR a IN (SELECT col1 FROM t3 WHERE col1 = c1))` returns one extra row here, where master throws INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND.
That sentence needs correcting, and it is worth settling in this PR whether a correlated leftover should raise 0A000 as well.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -439,7 +454,8 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), Some(finalJoinCond), joinHint)
introducedAttrs += exists
Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
Review Comment:
When `canRewrite` returns false the node is left as it is, and `transformDownWithPruning` then descends into its children. `SubqueryExpression.children` is `outerAttrs ++ joinCond`, so a subquery nested inside the leftover's own `joinCond` does get rewritten: an `ExistenceJoin` is grafted onto the subquery plan and an `exists` reference lands in that `joinCond`. The 0A000 message for a doubly nested shape shows it as `listquery(t2.c1, t2.c1, (t3.col1 = t2.c1), exists)`.
With default confs this shape ends at 0A000, so I have no repro where it alone yields a wrong answer. But in the paths where `PlanSubqueries` drops the whole `joinCond` (the NOT IN branch, or any branch with `decorrelatePredicateSubqueriesInJoinPredicate` off), that `ExistenceJoin` is computed with nothing referencing it. Worth confirming whether descending into a declined subquery is intended here.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -454,6 +470,70 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
}
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
Review Comment:
`referencesPlan`'s `e.references.intersect(plan.outputSet).nonEmpty` already exists in this object: the two lines in the `case j: Join` handler that test the left and right sides are the same expression doing the same job, classifying a subquery in a join condition by which side it references. With two copies, a later change to the classification is easy to make in one place only.
Reusing `referencesPlan` there would close it. Note that `PredicateHelper.canEvaluate` looks like the same test but uses `subsetOf`, so it is not a drop-in for either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4040734214
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -439,7 +454,8 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), Some(finalJoinCond), joinHint)
introducedAttrs += exists
Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
Review Comment:
`rewriteExistentialExprWithAttrs` now walks the expression with an explicit recursion, and a declined
sub-query is returned untouched, children included. With the default `canRewrite` the declined case
never matches, so the existing callers traverse exactly as before.
Worth noting for the record: with the correlated-leftover rejection from comment 1, every shape that
could reach this path now raises `0A000` first, so this change is no longer observable through SQL.
I kept it because relying on that coincidence to stay true is the kind of thing that breaks later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5750108267
Thanks for the update. I re-reviewed the current head and found one remaining regression in the new both-plans correlation guard; I will include the concrete simplification case in the review.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"conversation:5748836825","thread_id":"conversation:5748836825","verdict_sha256":"7cf6c1474d6825b4278e346f8cc2cad4f5a46d81e53f0050f359cdd21fb70a51"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4148316945
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) => collectOrDescend(not, in, truthOnly, reported)
+ case Not(child) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = false, reported)
+ case in: InSubquery => collectOrDescend(in, in, truthOnly, reported)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = true, reported))
+ sq.getOuterAttrs.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ case other =>
+ other.children.foreach(
Review Comment:
Right, and it was rejecting a valid query. `COALESCE(x, false)` is TRUE exactly where `x` is TRUE, so
only truth is observed and the existence-bit rewrite is exact; my generic fallback treated it as a
value position.
`Coalesce` whose next argument is the FALSE literal now passes the truth-only context to its first
argument. `COALESCE(c1 IN (...), false)` returns `1`, as PostgreSQL and DuckDB do, and is covered
alongside the non-foldable `If` and `CaseWhen` cases. A non-false default is not the same and stays
rejected: `COALESCE(c1 IN (...), true)` is TRUE where the IN is unknown and FALSE where it is FALSE,
and it is now in the rejected set next to `<=> false` and `IS NULL`. Removing the new arm fails the
test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4148365832
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
Review Comment:
Fixed. Each candidate's reference set is now built once, in `ownedBySubPlan` and
`referencesBothPlans`, and the rejection filter and the rewrite predicate share those rather than
recomputing `effectiveReferences` per question. The per-expression helper became a per-attribute
test, `referencedByOuterPlanOnly`, which is what the shared-ExprId rule actually needs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114386040
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,206 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) && !effectivelyReferencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
Review Comment:
Answering to both https://github.com/apache/spark/pull/58656#discussion_r4065967268 and this comment, Both are fixed. The rejection now reports the `Not` node, so `Not.sql` renders the user's NOT IN. The
arm is no longer behaviourally redundant, since it is what supplies that node. The contradictory
sentence in the `inSubqueryMayBeUnknown` scaladoc is gone, and the error message no longer says "The
IN subquery", which read wrongly for a NOT IN once the `Not` was preserved. Per the second half of
cloud-fan's note, the tests now assert `subqueryExpression` as well as the condition, including that
it starts with `(NOT ` exactly for the NOT IN predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4281445229
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala:
##########
@@ -161,4 +163,69 @@ class RewriteSubquerySuite extends PlanTest {
val existsExpected = LocalRelation(existsQuery.analyze.output).analyze
comparePlans(existsOptimized, existsExpected)
}
+
+ test("SPARK-59351: nested subquery of an EXISTS whose plan shares attributes with the outer " +
+ "plan") {
+ // The EXISTS arms route the nested subquery against the raw subquery plan, while the IN arms
+ // route it against the plan dedupSubqueryOnSelfJoin has already separated from the outer
+ // plan. A self-join can leave an ExprId shared between the two plans for the optimizer
+ // (SPARK-21835), and the analyzer is not involved here, so build that state directly: `a` is
+ // the same attribute in the outer and in the subquery plan.
+ val a = $"a".int
+ val outer = LocalRelation(a, $"b".int)
+ val sub = LocalRelation(a)
+ val nested = LocalRelation($"col1".int)
+ assert(outer.outputSet.intersect(sub.outputSet).nonEmpty,
+ "the two plans must share an attribute for this test to mean anything")
+
+ // The nested subquery references `a`, which only the subquery plan is meant to give it, and
+ // is correlated to it, so it must be rewritten against the subquery plan. Classifying `a` as
+ // a reference to the outer plan as well would report it as referencing both plans.
+ val nestedInSubquery = InSubquery(
+ Seq(a),
+ ListQuery(nested.select($"col1"), joinCond = Seq(EqualTo($"col1", a))))
+ val query = Filter(
+ Exists(sub, joinCond = Seq(Or(EqualTo($"b", a), nestedInSubquery))),
+ outer)
+
+ // The shared attribute is attributed to the subquery plan, so the nested subquery is routed
+ // there rather than reported as referencing both plans. What then reports this state is the
+ // conflict check that dedupSubqueryOnSelfJoin has always run over the semi join condition,
+ // which names the duplicated attribute: an attribute shared between the two plans that the
+ // hoisted condition mentions cannot be turned into a semi join whatever the routing does.
+ val e = intercept[AnalysisException](Optimize.execute(query))
+ assert(e.getCondition == "_LEGACY_ERROR_TEMP_1212",
+ s"expected the pre-existing conflict check to report this, got ${e.getCondition}")
+ }
+
+ test("SPARK-59351: the rewrite keeps the origin and the tags of the sub-query it replaces") {
+ // The rewrite of a nested existential sub-query builds its replacement by hand rather than
+ // through TreeNode.transformDownWithPruning, so it has to reproduce what that does: apply the
+ // rule under the origin of the node being replaced and copy its tags. Without this a runtime
+ // error raised from the rewritten condition quotes the enclosing operator instead of the
+ // sub-query fragment, which no result or plan assertion would notice.
+ val outer = LocalRelation($"a".int)
+ val inner = LocalRelation($"x".int)
+ val tag = TreeNodeTag[String]("SPARK-59351")
+ val origin = Origin(line = Some(42), startPosition = Some(7))
+ val a = outer.output.head
+ val x = inner.output.head
+ // Correlated, so that the rule's unary node handler picks the projection up at all.
+ val exists = CurrentOrigin.withOrigin(origin) {
+ Exists(inner, outerAttrs = Seq(a), joinCond = Seq(EqualTo(a, x)))
+ }
+ exists.setTagValue(tag, "kept")
+
+ // An Exists outside a Filter goes through the unary node handler, which is the path that
+ // builds the replacement by hand.
+ val rewritten = RewritePredicateSubquery(Project(Seq(Alias(exists, "e")()), outer))
+ val replacements = rewritten.expressions.flatMap(_.collect {
+ case attr: AttributeReference if attr.name == "exists" => attr
+ })
+ assert(replacements.size == 1, s"expected one exists attribute in\n$rewritten")
+ assert(replacements.head.origin == origin,
Review Comment:
Confirmed: the new test pins the origin of every node synthesized for IN and NOT IN, and it checks that the reused hoisted condition keeps its own origin.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4257086416","thread_id":"inline:4257086416","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,426 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: nested IN subquery whose result can be unknown") {
+ withTempView("t1", "t2", "t2n", "t3n", "t3", "t4", "tb") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // A nullable compared value, against a subquery column that is not nullable.
+ Seq(Some(1), None).toDF("c1").persist().createOrReplaceTempView("t2n")
+ Seq((1, 1), (9, 9)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ // A boolean relation, to compare the result of a nested IN subquery against.
+ Seq(false).toDF("b").persist().createOrReplaceTempView("tb")
+ // The NULL matches no value of t2, so c1 IN (SELECT col1 FROM t3n) is unknown there, and
+ // c1 NOT IN (SELECT col1 FROM t3n) is never true.
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // An existence join only records whether a row matched, so its exists attribute is false
+ // where IN is unknown. That is not observable while the value feeds a predicate, but it is
+ // once it reaches anything else, so these are rejected rather than rewritten. Both
+ // configurations are checked, as the rejection must not depend on the rewrite of predicate
+ // subqueries in join conditions.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> false",
+ "(c1 IN (SELECT col1 FROM t3n)) IS NULL",
+ // COALESCE to a non-false default does tell unknown from false.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), true)",
+ "(c1 NOT IN (SELECT col1 FROM t3n)) <=> false").foreach { predicate =>
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)")
+ .collect()
+ }
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT")
+ // A NOT IN must be reported as a NOT IN, not as the IN underneath it.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.startsWith("(NOT ") == predicate.contains("NOT IN"),
+ s"unexpected subqueryExpression for $predicate: $reported")
+ }
+ }
+ }
+
+ // The ordinary case: the value of the nested IN feeds a predicate, where unknown and
+ // false cannot be told apart, so it is still rewritten even though it can be unknown.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t3n))""".stripMargin,
+ Row(1) :: Nil)
+
+ // And a nested IN that cannot be unknown is rewritten even outside a predicate.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3)) <=> false))""".stripMargin,
+ Row(1) :: Row(2) :: Row(3) :: Nil)
+
+ // Only truth is observed by `<=> true`, by an IF predicate and by a CASE WHEN
+ // condition, so those are rewritten rather than rejected. SimplifyConditionals folds
+ // the latter two into the first, so ordinary SQL reaches this position.
+ //
+ // The result branches of a conditional are consumed wherever the conditional is, so
+ // under a filter predicate they are truth-only as well. The last two cases do not
+ // fold, so they exercise that inheritance rather than the rewrites above.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> true",
+ "IF(c1 IN (SELECT col1 FROM t3n), true, false)",
+ "CASE WHEN c1 IN (SELECT col1 FROM t3n) THEN true ELSE false END",
+ "IF(c1 > 0, c1 IN (SELECT col1 FROM t3n), false)",
+ "CASE WHEN c1 > 0 THEN c1 IN (SELECT col1 FROM t3n) ELSE false END",
+ "CASE WHEN c1 > 100 THEN false ELSE c1 IN (SELECT col1 FROM t3n) END",
+ // COALESCE to false maps unknown and false alike, so only truth is observed.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), false)")
+ .foreach { predicate =>
+ checkAnswerAndPlan(
+ s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)",
+ Row(1) :: Nil)
+ }
+
+ // The unknown can come from the compared value rather than from the subquery column.
+ // spark.sql.legacy.inSubqueryNullability makes InSubquery.nullable ignore the
+ // nullability of the list query columns, but it always keeps that of the compared
+ // values, and the guard re-derives both, so this case is rejected under either value.
+ Seq("true", "false").foreach { legacyNullability =>
Review Comment:
Agreed, and thanks for checking it end to end. Under legacy=true the `<=> false` is simplified to NOT IN and takes the null-aware rewrite, so pinning rejection under the default and the correct answer under legacy is the right assertion.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4257086424","thread_id":"inline:4257086424","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,476 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. The two plans share no ExprId in any
+ * shape reached by the suite: `DeduplicateRelations` has given the sub-query its own, and the
+ * EXISTS arms classify against the raw sub-query plan while the IN arms classify against the
+ * plan `dedupSubqueryOnSelfJoin` has already separated from the outer plan. The order is
+ * therefore defensive rather than something a query exercises, and it is the safe one: a
+ * nested sub-query resolves in the scope of the sub-query it is nested in, so an attribute
+ * both plans produce is one of that sub-query's own, and attributing it to `outerPlan` instead
+ * would let a single shared attribute satisfy both sides of the classification and have a
+ * nested sub-query referencing only the inner query rejected as referencing both.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite
+ * without rejecting the sub-query, as a sub-query referencing neither plan can be left in the
+ * join condition unharmed.
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(sq)) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(ownedBySubPlan)
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) =
+ rewriteExistentialExprWithAttrs(conditions, subPlan, ownedBySubPlan)
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ // COALESCE(x, false) maps both unknown and FALSE to FALSE, so only truth is observed. A
+ // different default does not: COALESCE(x, true) is TRUE where x is unknown and FALSE where x
+ // is FALSE.
+ case Coalesce(child +: rest) if rest.headOption.contains(Literal.FalseLiteral) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) => collectOrDescend(not, in, truthOnly, reported)
+ case Not(child) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = false, reported)
+ case in: InSubquery => collectOrDescend(in, in, truthOnly, reported)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = true, reported))
+ sq.getOuterAttrs.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ case other =>
+ other.children.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ }
+
+ /**
+ * Appends `report` to `reported` if the IN sub-query `in` would lose an observable unknown, and
+ * otherwise descends into the values it compares and into its list query. The descent matters
+ * because [[InSubquery]] is a [[Predicate]] rather than a [[SubqueryExpression]], so an IN
+ * sub-query nested in the values or in the hoisted join condition of another one is only reached
+ * from here.
+ */
+ private def collectOrDescend(
+ report: Expression,
+ in: InSubquery,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = {
+ if (!truthOnly && inSubqueryMayBeUnknown(in)) {
+ reported += report
+ } else {
+ in.values.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ collectUnknownSensitiveInSubqueries(in.query, truthOnly, reported)
+ }
+ }
+
+ /**
+ * Returns true if the row comparison of the IN sub-query `in` can evaluate to unknown, that is
+ * if it can return NULL rather than only TRUE or FALSE. An existence join cannot represent that
+ * third value, see [[rewriteExistentialExprInJoinCondition]].
+ *
+ * This deliberately does not use `InSubquery.nullable`, which under
+ * `spark.sql.legacy.inSubqueryNullability` (SPARK-43413) reports only the nullability of the
+ * compared values and ignores that of the list query's columns. What is protected here is the
+ * three-valued result the rewrite would drop, which does not depend on that configuration, so
+ * the nullability is re-derived from both sides.
+ */
+ private def inSubqueryMayBeUnknown(in: InSubquery): Boolean = {
+ // Iterators, so a wide row-valued IN stops at the first nullable side rather than pairing up
+ // the whole arity first.
+ in.values.iterator.zip(in.query.childOutputs.iterator).exists {
Review Comment:
Confirmed: the row-valued cases put nullability only in the second pair, once on each side. Both are rejected, and the fully non-nullable shape is still rewritten.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4257086431","thread_id":"inline:4257086431","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4274594390
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,412 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `attr` is produced by `outerPlan` and not by `subPlan`.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`.
+ * This matters when the two share ExprIds, which a self-join can leave for the optimizer to
+ * deduplicate (SPARK-21835): the EXISTS arms classify against the raw sub-query plan, while the
+ * IN arms classify against the plan that `dedupSubqueryOnSelfJoin` has already separated from
+ * the outer plan. Without this a single shared attribute would satisfy both sides of the
+ * classification, so a nested sub-query that references only the sub-query it is nested in
+ * would be taken for one referencing both plans and rejected. A nested sub-query resolves in
+ * the scope of that sub-query, so a shared attribute is one of its own.
+ */
+ private def referencedByOuterPlanOnly(
+ attr: Attribute,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): Boolean = {
+ outerPlan.outputSet.contains(attr) && !subPlan.outputSet.contains(attr)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
Review Comment:
Addressed on the current HEAD — apologies for the late reply. This also closes `r4139416496` from the round before, which I had marked addressed when it was not.
Both halves of this check out, and both are fixed.
**The flattened sequence.** `AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))` is worse than it looks: the `flatMap` builds a `Seq[Attribute]` as long as the total number of references, and then `AttributeSet.apply(Iterable[Expression])` is `fromAttributeSets(baseSet.map(_.references))`, so each of those attributes gets its own single-element `AttributeSet` allocated purely to be folded away. `effectiveReferences` now merges the child sets directly:
```scala
AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
```
**The repeated reconstruction and the separate scans.** The same candidate was classified up to three times once by the both-plan rejection, once by the UNKNOWN filter, once by the rewrite's `canRewrite` each time rebuilding the set and scanning it twice, once for `subPlan` and once for `referencedByOuterPlanOnly`. `referencedByOuterPlanOnly` is replaced by `planReferences`, which answers both questions in a single pass with an early exit once both plans are known to be referenced, and the result is memoised per candidate:
```scala
val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
```
Identity keying is deliberate: the candidates are nodes of the expressions being classified, so structural comparison would compare the sub-query plans they hold.
Per candidate this goes from 3 set constructions and up to 6 scans to 1 and 1.
**On the collision ordering**, which is the compatibility half of your request. The single-pass form probes `subPlan` first, preserving the "a shared attribute belongs to the sub-query" tie-break that `referencedByOuterPlanOnly` had. I first wrote in the scaladoc that no shape in the suite reaches that branch that was wrong, and you caught it in `r4257086406`: the shared-attribute fixture in `RewriteSubquerySuite` does reach it. The comment has since been corrected to distinguish analyzer-produced SQL plans, which do not reach it because `DeduplicateRelations` has already separated the ExprIds, from that optimizer-level fixture, which builds the overlap directly. It pins the shape but not the outcome of the ordering, since `dedupSubqueryOnSelfJoin`'s pre-existing conflict check reports the duplicated attribute first either way.
The other shapes you named — inner-only, outer-only, both-plan, unreferenced and simplified-away-correlation are covered by the existing cases in the two `SPARK-59351` tests, and all pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4065967215
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -414,7 +433,8 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
existenceJoin, newCondition, subHint)
introducedAttrs += exists
exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
val exists = AttributeReference("exists", BooleanType, nullable = false)()
// Deduplicate conflicting attributes if any.
val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
Review Comment:
The deduplication anchor moves from the outer plan to the sub-query plan on the new route, and that has two consequences.
Before this PR `newPlan` here was always the outer plan, and `PullupCorrelatedPredicates` had already deduplicated the nested sub-query's plan against exactly that node (`decorrelate(sub, plan, ...)` at L837/L845; `DecorrelateInnerQuery.scala:468` derives `outputPlanInputAttrs` from `outerPlan.inputSet`). In phase 1 `newPlan` is the sub-query plan, against which nothing has deduplicated.
**(a) ids shared with the outer plan are no longer aliased, and `buildJoin` cannot compensate.** `Join.computeOutput` for `ExistenceJoin` is `leftOutput :+ j.exists` (`basicLogicalOperators.scala:729-730`), so `duplicates = outerRefs.intersect(subplan.outputSet)` at L103 never sees the nested relation below the new join. For
```sql
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT a FROM t1))
```
the result is `Join(t1, Join(t2, t1', ExistenceJoin), LeftSemi)` where `t1'` can reuse the outer `t1`'s exprIds across sibling subtrees — a position no `duplicateResolved` check inspects, since that predicate compares only one join's two children (L820). `LogicalPlanIntegrity.checkIfSameExprIdNotReused` would fire under plan-change validation (default `Utils.isTesting`, `SQLConf.scala:706`).
**(b) The `Exists` arm can now throw instead of aliasing.** It reaches dedup through `buildJoin` (L432), which passes the condition (L65), and `dedupSubqueryOnSelfJoin` throws `conflictingAttributesInJoinConditionError` when the condition references a duplicate (L105-111). A nested correlated `EXISTS`'s `newCondition` always references sub-plan attributes, so an id collision with the sub-query plan surfaces as a user-facing `CONFLICTING_ATTRIBUTES_IN_JOIN_CONDITION`. Note this line and L466 call the helper with no `condition`, so the IN arms only alias and never throw — an asymmetry inside the same helper.
`DeduplicateRelations` does descend into `subquery.plan` (`DeduplicateRelations.scala:249-253`) and resolves many of these in the analyzer, but the SPARK-21835 tests at `SubquerySuite.scala:1128-1190` exist because conflicts still reach the optimizer. There is no test here where the nested relation overlaps either plan.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
Review Comment:
Both items from the previous review round look like they were answered but not pushed.
The last commit on this branch is `3e656616ced`, committed `2026-09-21T16:49:46Z`. The two replies are `16:57:33Z` (on `:565`) and `16:58:23Z` (here), i.e. eight minutes *after* the commit, and neither change is in the tree:
1. The reply on `:565` says `isCorrelated` was replaced by `hasCorrelatedCondition` reading the hoisted join condition, and that routing now uses the surviving state. `grep -n "hasCorrelatedCondition\|effective\|surviving"` on this file returns nothing; `isCorrelatedSubquery` (L634-638) still reads `listQuery.isCorrelated`, `referencesPlan` (L486-488) still reads `e.references`, and there is no `false AND col1 = c1` regression test. The 0A000 regression against 4.0.1 reproduced in that reply is therefore still live.
2. This line still reads `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`, without the `.enabled` suffix that reply says was added. That is not the registered key.
The PR description also still contains "Both the routing and the rejections read the hoisted condition and the compared values rather than `references` and `isCorrelated` for this reason", which describes the unpushed version rather than this code.
CI being green does not contradict any of this: nothing covers either item. Could you check whether a commit is missing?
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) && referencesPlan(sq, subPlan) &&
+ referencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => referencesPlan(sq, subPlan) && !referencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => referencesPlan(e, subPlan) && !referencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ case Not(child) => unknownSensitiveInSubqueries(child, inPredicate)
+ case in: InSubquery =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ // The join condition of a sub-query expression is a predicate, evaluated by the join that
+ // the sub-query is rewritten into.
+ case sq: SubqueryExpression =>
+ sq.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ case other =>
Review Comment:
This fallback treats every parent other than `And`/`Or`/`Not` as a value position, which over-rejects shapes where the rewrite is exact.
```sql
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2 WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3n)) <=> true))
```
with a NULL in `t3n` is rejected, but it should not be: `unknown <=> TRUE` and `FALSE <=> TRUE` are both FALSE, and the existence join's condition is the NULL-rejecting `values.zip(newSub.output).map(EqualTo.tupled)` (L467), so `exists` is TRUE exactly when IN is TRUE. `ReplaceNullWithFalseInPredicate` already lists this position among those where NULL and FALSE are interchangeable (L79-82, with the comment "whether the other side is null or false has no difference"), together with `If`'s predicate and `CaseWhen` branch conditions — none of which this helper recognises.
The shape survives folding for precisely the queries this PR targets: `SimplifyBinaryComparison`'s `case a EqualNullSafe TrueLiteral if !a.nullable => a` (`expressions.scala:620`) is guarded by `!a.nullable`, which fails exactly when the IN is nullable; and `SimplifyConditionals` (`expressions.scala:644, 655`) normalises `IF(c1 IN (...), true, false)` and `CASE WHEN c1 IN (...) THEN true ELSE false END` into that same rejected `EqualNullSafe`. So ordinary SQL lands here, and `<=> true` is rejected while the new test rejects `<=> false` legitimately — an inconsistent pair.
Reusing the case list from `ReplaceNullWithFalseInPredicate` would close it. (Its lambda-body cases are not reachable from this rule, so they are not part of this.)
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) && referencesPlan(sq, subPlan) &&
+ referencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => referencesPlan(sq, subPlan) && !referencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => referencesPlan(e, subPlan) && !referencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
Review Comment:
Two things here.
**The error names the wrong operator.** This returns `in`, the un-negated `InSubquery`, so `nestedInSubqueryWithUnknownResultError` calls `.sql` on it and `InSubquery.sql` is `s"(${value.sql} IN (${query.sql}))"` (`predicates.scala:424`) — the user's `NOT` disappears and the message reads "The IN subquery (c1 IN (...))" for a `NOT IN`. The new test feeds exactly this shape, `(c1 NOT IN (SELECT col1 FROM t3n)) <=> false`, but asserts only on `getCondition`, so nothing catches it. Returning the `Not` node renders correctly via `Not.sql`.
**The arm is behaviourally redundant.** Deleting it changes nothing: `case Not(child)` on the next line falls through to `case in: InSubquery` and returns the identical value. It exists only to carry the comment, and the duplicated condition can drift.
Separately, the scaladoc on `inSubqueryMayBeUnknown` (L620-621) says "NOT IN is excluded", which contradicts both this call site and the comment directly above it.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
Review Comment:
Also minor: this adds two full, unpruned traversals of the hoisted conditions for every correlated predicate sub-query, including the common case where there is no nested sub-query at all.
Only the third step, `rewriteExistentialExprWithAttrs`, prunes on tree-pattern bits (L423). This `collect` uses `TreeNode.collect`, which `foreach`es every node, and `unknownSensitiveInSubqueries` (L574) recurses to the leaves through `case other => other.children.flatMap(...)` with no pattern check anywhere, allocating a result collection at each interior node even when everything comes back empty.
For an ordinary correlated EXISTS or IN with no nested sub-query — TPC-H Q21, TPC-DS q10/q35, most real workloads — the pre-PR code did one `O(1)` bit test per hoisted condition; now the whole condition tree is walked twice, per predicate sub-query, per compile. The IN arm passes `inConditions ++ conditions` (L177), so a K-column IN's synthesised equalities are walked twice as well.
A guard at the top of this method is semantically exact, since both collectors match only `Exists` and `InSubquery`:
```scala
if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
return (conditions.reduceOption(And), subPlan)
}
```
The same check at the head of `unknownSensitiveInSubqueries` additionally prunes sub-query-free branches of an `OR`.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +158,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
Review Comment:
The four arms hand the router differently-prepared sub-plans.
This arm and the `Not(Exists)` arm below pass the raw `sub`; the `InSubquery` and `Not(InSubquery)` arms first compute `val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))` (L174, L190) and route against that, which by construction no longer overlaps `p.outputSet`. For EXISTS the dedup happens only later, inside `buildJoin` (L65).
So whenever `sub.outputSet ∩ p.outputSet` is non-empty — exactly what L103 tests for, and the shape of the SPARK-21835 self-join tests at `SubquerySuite.scala:1128` — a single referenced attribute satisfies *both* `referencesPlan(sq, subPlan)` and `referencesPlan(sq, outerPlan)` at L565-566. A correlated nested sub-query that genuinely references only one side is then rejected with `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY`, whose wording does not describe the query; an uncorrelated one is declined by both passes and stranded in the join condition, where the default-configuration handler rejects it at L248-251.
Deduplicating `sub` before this call, as the IN arm already does, would remove the asymmetry.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
Review Comment:
Phase 1 accepts on a positive test, `referencesPlan(e, subPlan) && !referencesPlan(e, outerPlan)` (L581), but phase 2 accepts on the *negation*, `!referencesPlan(e, subPlan)`. The two are not complements, and the gap is filled in the unsafe direction: whatever `references` cannot see is not left alone, it is affirmatively grafted onto the outer plan.
`SubqueryExpression.references` is `AttributeSet.fromAttributeSets(nonOuterScopeAttrs.map(_.references))` (`expressions/subquery.scala:88-89`) — it drops every outer attribute containing an `OuterScopeReference` (partitioned at `:85-86`) and never includes the sub-query's own `joinCond`. A nested sub-query whose link to the sub plan is invisible to `references` therefore fails phase 1's positive test, passes this negation, and gets an `ExistenceJoin` on the outer plan whose condition references an attribute neither child produces — the exact invalid-plan shape SPARK-59351 fixes.
I could not establish reachability (nested correlation is rejected earlier today), so this is a robustness point rather than a live bug. But the routing's correctness currently rests on an unstated invariant — that `references` covers every attribute the rewrite will put in a join condition — and a positive test here (`references the outer plan, or references neither`) would fail safe instead.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) && referencesPlan(sq, subPlan) &&
+ referencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => referencesPlan(sq, subPlan) && !referencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => referencesPlan(e, subPlan) && !referencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ case Not(child) => unknownSensitiveInSubqueries(child, inPredicate)
+ case in: InSubquery =>
Review Comment:
This arm and the `Not(InSubquery)` arm above return without recursing into `in.children` (`values :+ query`), while the `SubqueryExpression` arm three lines below explicitly does descend. `InSubquery` is a `Predicate`, not a `SubqueryExpression`, so it can never reach that arm.
```sql
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2 WHERE a = c1
OR c1 IN (SELECT col1 FROM t4 WHERE k = c1 AND ((c1 IN (SELECT col1 FROM t3n)) <=> false)))
```
The inner `<=> false` is a level-2 correlated predicate and is hoisted into the level-2 `ListQuery`'s `joinCond`. The walk stops at the level-2 `InSubquery` with `inPredicate = true` and never examines the `t3n` IN. Phase 1 then rewrites the level-2 sub-query onto the sub plan, L468 puts the un-inspected `EqualNullSafe` into the new `ExistenceJoin` condition, and the `case j: Join` handler — which has no unknown guard either — turns it into a second existence join whose `exists` is FALSE where the IN is unknown. Before this PR that query produced the missing-input plan this PR fixes, so the path moves from a broken plan to a silent wrong answer. The `values` side has the same hole.
This is also inconsistent with the `referencingBothPlans` collector at L563-567, whose comment states that it "walks the whole expression, including the join condition of a sub-query that the rewrite below declines to descend into".
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,14 +405,26 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
Review Comment:
Replacing `transformDownWithPruning` with a hand-rolled recursion drops the two things that traversal did besides walking the tree.
`TreeNode.transformDownWithPruning` applies the rule inside `CurrentOrigin.withOrigin(origin)` (`TreeNode.scala:510-512`) and calls `afterRule.copyTagsFrom(this)` when a node is replaced (`TreeNode.scala:525`). The new `rewrite` does neither, so every node the three arms build — the `AttributeReference("exists")` at L428/438/464, the `EqualTo`/`Or`/`IsNull`/`And` join conditions, and the `Join`s at L432/457/470 — inherits the enclosing `Filter`'s ambient origin instead of the sub-query expression's, and `TreeNodeTag`s on the replaced `Exists`/`InSubquery` are lost. The visible effect is the `SQLQueryContext` on runtime errors raised from the rewritten join condition: a `DIVIDE_BY_ZERO` inside a hoisted predicate now quotes the whole `Filter` rather than the sub-query fragment.
Worth noting that this cost lands on code paths the PR does not otherwise touch: `handleUnaryNode` (L347) routes every `UnaryNode` through here, and with the default `canRewrite = _ => true` the new do-not-descend arm at L474-475 is unreachable for those callers. Wrapping the arm bodies in `CurrentOrigin.withOrigin(expr.origin)` and adding `copyTagsFrom` restores it. Pruning and gc-churn avoidance are preserved correctly otherwise.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) && referencesPlan(sq, subPlan) &&
+ referencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => referencesPlan(sq, subPlan) && !referencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => referencesPlan(e, subPlan) && !referencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ case Not(child) => unknownSensitiveInSubqueries(child, inPredicate)
+ case in: InSubquery =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ // The join condition of a sub-query expression is a predicate, evaluated by the join that
+ // the sub-query is rewritten into.
+ case sq: SubqueryExpression =>
Review Comment:
`sq.children` is `outerAttrs ++ joinCond` (`expressions/subquery.scala:91`), but the comment above justifies `inPredicate = true` only for the join condition. `outerAttrs` are value expressions, not predicates.
So an IN sub-query reachable through an `outerAttrs` entry — e.g. a compound correlated expression that `PullupCorrelatedPredicates` lifts into `outerAttrs`, which `getOuterReferences` explicitly supports via `stripOuterReference(a)` for aggregate expressions (`expressions/subquery.scala:236-237`) — is marked as a predicate position and skipped by the unknown-result check although its value is not tested for TRUE. Same failure class as the other predicate-context gaps: a three-valued IN becomes a non-nullable `exists` bit and the answer changes.
Passing `inPredicate = true` only for the `joinCond` slice would match what the comment already says.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) && referencesPlan(sq, subPlan) &&
+ referencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => referencesPlan(sq, subPlan) && !referencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => referencesPlan(e, subPlan) && !referencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ case Not(child) => unknownSensitiveInSubqueries(child, inPredicate)
+ case in: InSubquery =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ // The join condition of a sub-query expression is a predicate, evaluated by the join that
+ // the sub-query is rewritten into.
+ case sq: SubqueryExpression =>
+ sq.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ case other =>
+ other.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = false))
+ }
+
+ /**
+ * Returns true if `e` is a positive IN sub-query whose row comparison can evaluate to unknown,
+ * that is one that can return NULL rather than only TRUE or FALSE. An existence join cannot
+ * represent that third value, see [[rewriteExistentialExprInJoinCondition]]. NOT IN is excluded,
+ * as [[rewriteExistentialExprWithAttrs]] gives it a null-aware join condition of its own.
+ */
+ private def inSubqueryMayBeUnknown(e: Expression): Boolean = e match {
Review Comment:
This re-derives `InSubquery.nullable` (`predicates.scala:414-423`), which is already `values.exists(_.nullable) || query.childOutputs.exists(_.nullable)`, but without its `LEGACY_IN_SUBQUERY_NULLABILITY` branch (`SQLConf.scala:7740`, SPARK-43413).
I think the config-independence is the *right* call here — the three-valued runtime behaviour being protected does not depend on a flag, and reusing `in.nullable` would make the check unsound under it. But nothing says so and no test pins it, and the consequence is easy to miss: under `spark.sql.legacy.inSubqueryNullability=true` the new guard quietly stops covering two of its own three test predicates.
- `(c1 IN (...)) IS NULL` is folded to false by `NullPropagation`'s `case IsNull(c) if !c.nullable` (`expressions.scala:940`), so the IN leaves the plan entirely.
- `(c1 IN (...)) <=> false` becomes `Not(InSubquery)` via `SimplifyBinaryComparison`'s `case a EqualNullSafe FalseLiteral if !a.nullable => Not(a)` (`expressions.scala:622`), which this helper then classifies as `inPredicate` and does not reject.
A comment stating the divergence is deliberate, plus a test under that flag, would keep it from being read as an oversight. `listQuery.childOutputs` would also be the intended accessor rather than `plan.output` (the `zip` truncation makes them equal today, but `ListQuery` carries `numCols` precisely because `plan.output` grows after decorrelation).
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
Review Comment:
Minor, but this is on a hot path and the fix is free: the operands are the wrong way round.
`AttributeSet.intersect` is `new AttributeSet(other.baseSet.filter(baseSet.contains))` (`expressions/AttributeSet.scala:144-145`) — it iterates and materialises from the *argument*. So `e.references.intersect(plan.outputSet)` walks all of `plan.outputSet` and builds a `LinkedHashSet`, making each call `O(|plan.output|)` rather than `O(|e.references|)`, for a boolean that `.nonEmpty` immediately discards.
`referencesPlan` is called two to four times per nested sub-query (L565-566, L576, L581, L541) plus twice per sub-query in the `Join` branch at L239-240. Over a 300-column relation that is roughly 1,200 hash probes and four set allocations per nested sub-query where a handful would do.
`plan.outputSet.intersect(e.references).nonEmpty` iterates the small side. Since this helper now also backs L239-240, one edit improves both sites.
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3227,262 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // Checks the result, and that every node of the optimized plan can produce the attributes
+ // it references. The latter is what this fix is about: an existence join whose condition
+ // references an attribute produced by neither of its children can still return the right
+ // answer when the nested relation is empty at runtime, because the invalid condition is
+ // then never bound. checkAnswer alone would not catch it, as the missing input checks it
+ // runs only look at the root of the plan.
+ def checkAnswerAndPlan(query: String, expected: Seq[Row]): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ checkAnswer(df, expected)
+ }
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // would drop its correlated predicate col1 = c1 and return an extra row, so it is rejected.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ val df = sql(uncorrelated)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
Review Comment:
This assertion cannot see what this test exists to exercise.
`QueryPlan.references` is `AttributeSet(expressions) -- producedAttributes` (`plans/QueryPlan.scala:139-141`), and `AttributeSet(exprs)` folds `_.references`, which for a `SubqueryExpression` is only its `outerAttrs` — never its `joinCond`. Separately `LogicalPlan.collect` walks `children`, which never include a sub-query expression's plan (hence the separate `QueryPlan.subqueriesAll`). So for any node still carrying a predicate sub-query in its condition, `missingInput` sees neither that sub-query's join condition nor anything inside its plan.
That is exactly the `uncorrelated` case here, whose whole point is that the `InSubquery` *survives* in the join condition. It passes today only because that particular sub-query is uncorrelated and its `joinCond` is empty.
Two smaller points in the same area:
- The block is a local `def checkAnswerAndPlan` at L3253-3262, copied verbatim here, and absent from the third test (L3442-3487), whose two positive cases call only `checkAnswer`. Those are the shapes where — as the comment at L3366-3374 explains — an invalid `ExistenceJoin` still returns the right answer when the nested relation is empty at runtime, so a regression there trips nothing. Lifting the helper to a suite-level `private def` and using it in all three would fix that.
- None of the three tests asserts plan *shape*, so a fix that placed the `ExistenceJoin` on the other side would pass identically. Asserting the join sits on the sub-query side would pin the actual contract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4380469884
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothInputs.toSeq)
+ }
+ val unknownResult = carried.filter(sq => referencesOneInput(sq) && observesUnknown(sq))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult.toSeq)
+ }
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ carried.filter(sq => referencesOneInput(sq) && isInput(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) = rewriteExistentialExprWithAttrs(Seq(sq), p)
+ newCondition = newCondition.transform {
Review Comment:
Confirmed and fixed in 2666748431d. `rewriteCarriedExistentialExprs` now routes the NOT IN units before the bare ones (stable sort on `Not(_: InSubquery)`), so the bare IN's `fastEquals` substitution no longer rewrites the IN inside the `Not`. The null-aware existence join built for the `Not` unit is now the one the condition references, and the bare IN gets its own existence join, the same as when the query is written with two distinct IN sub-queries.
Added `SPARK-59351: a carried IN shared between a bare and a negated position` with your repro (t1(a) = 1, -1, NULL; t2(c1) = 1; t3(col1) = 3, NULL), run under both values of `decorrelatePredicateSubqueriesInJoinPredicate.enabled`. It returns only a = 1. The test selects `s.a` only: the projected `b` column on that route is the pre-existing two-valued projection behaviour tracked by SPARK-59862, so it is not asserted here.
As you noted, the `case j: Join` arm has the same `fastEquals` pattern at the merge base; I have left that arm alone as pre-existing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4380528074
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -184,8 +188,20 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Use EXISTS if performance matters to you.
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ // The hoisted correlated predicates may carry existential sub-queries that only the
+ // sub-query plan can evaluate, see rewriteExistentialExprInJoinCondition.
+ // This arm returns a bare Join, which this rule does not offer to its own handling of
Review Comment:
Done. The production comment now states the same condition as the test: keeping the residual holds only while this NOT IN is the last sub-query conjunct. Otherwise the fold wraps this anti join as `p` of a later EXISTS or IN arm in `Project(p.output, Join(p, ...))`, the top-down pass reaches it in the `case j: Join` arm, and under the default configuration that arm rewrites the residual into a two-valued `exists`. That outcome is pre-existing and is left to SPARK-59862. I removed the "both correct" and "were it ever wrapped" wording.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4356971327
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,625 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. With EXISTS the rewrite returns a Project over the join, which that handling
+ // reaches under the default configuration and rejects with
+ // UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION instead, so the query would still
+ // fail but on a different condition than the one asserted here.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: existential subquery carried by a rewritten nested subquery") {
+ // The nested IN over t4 is owned by the subquery plan, so it is rewritten into an existence
+ // join and the condition PullupCorrelatedPredicates hoisted into it is grafted onto that
+ // join. That condition carries a third-level IN correlated to t4, which references neither
+ // the outer nor the subquery plan and so is routed by neither of the guards above. It has to
+ // be rewritten against the inputs of the join just built: left in place, PlanSubqueries plans
+ // it as an in-subquery filter and drops its own correlated predicate col1 = k, which returns
+ // every row of t1 instead of the one row where the correlation holds.
+ withTempView("t1", "t2", "t4", "t5") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ // No k equals any c1, so only the third-level IN can satisfy the hoisted disjunction, and
+ // its correlated predicate col1 = k is false for every row of t4: dropping it makes the
+ // level-2 IN hold for c1 = 1 and returns 1, 2 and 3.
+ Seq((1, 100), (9, 7)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(1).toDF("col1").persist().createOrReplaceTempView("t5")
+ val query =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t4
+ | WHERE k = c1 OR col1 IN (SELECT col1 FROM t5 WHERE col1 = k)))""".stripMargin
+ // The routing must not depend on the rewrite of predicate subqueries in join conditions,
+ // which is what used to make these two configurations disagree: it reached the grafted
+ // condition only when enabled.
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswerAndPlan(query, Row(1) :: Nil)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: NOT IN keeps an uncorrelated three-valued residual") {
+ // The NOT IN rewrite returns a bare Join, which this rule does not offer to its own handling
+ // of predicate subqueries in join conditions, so an uncorrelated nested IN left in the
+ // anti-join condition stays there and InSubqueryExec evaluates it three-valued. No existence
+ // join is built for it and it has no hoisted condition to lose, so it must not be rejected:
+ // these queries returned the standard SQL answer before SPARK-59351 and must keep doing so.
+ withTempView("t1", "t2", "t3n", "t4n") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ // The NULL matches no c1, so c1 IN (SELECT col1 FROM t3n) is unknown for every row of t2.
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // Nullable col1 again, but correlated through k = c1.
+ Seq((Some(3), 1), (None, 8)).toDF("col1", "k").persist().createOrReplaceTempView("t4n")
+
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> false",
+ "(c1 IN (SELECT col1 FROM t3n)) IS NULL",
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), true)",
+ "(c1 NOT IN (SELECT col1 FROM t3n)) <=> false").foreach { predicate =>
+ // Every such predicate is FALSE for every row of t2, so the subquery reduces to a = c1
+ // and only 2 and 3 survive the NOT IN. Rejecting instead, or rewriting the nested IN into
+ // a two-valued exists, would change this.
+ checkAnswer(
+ sql(s"SELECT * FROM t1 WHERE a NOT IN (SELECT c1 FROM t2 WHERE a = c1 OR $predicate)"),
Review Comment:
Both points are right, and the comment was worse than you describe: I wrote that every predicate is FALSE for every row of t2, and two of the four are TRUE there, since `unknown IS NULL` is TRUE and `COALESCE(unknown, true)` is TRUE.
Rebuilt on your suggestion, with `c1 > a` instead of the key equality, which makes every case discriminate and splits them into two groups by what unknown does:
| predicate | three-valued (asserted) | two-valued |
|---|---|---|
| `(c1 IN (...)) <=> false` | `1, 2, 3` | `2, 3` |
| `(c1 NOT IN (...)) <=> false` | `1, 2, 3` | `2, 3` |
| `(c1 IN (...)) IS NULL` | `2, 3` | `1, 2, 3` |
| `COALESCE(c1 IN (...), true)` | `2, 3` | `1, 2, 3` |
All four asserted answers match DuckDB. Checked that they discriminate against a two-valued rewrite specifically, not only against a rejection: letting `canRewrite` consume the residual while keeping it out of the rejection makes the test fail on the answers.
The truth-only case at the end now uses `checkAnswerAndPlan`, so "rewritten, not kept" is actually checked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5601633301
@cloud-fan @dongjoon-hyun could you PTAL? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114361290
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) && referencesPlan(sq, subPlan) &&
+ referencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => referencesPlan(sq, subPlan) && !referencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => referencesPlan(e, subPlan) && !referencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ case Not(child) => unknownSensitiveInSubqueries(child, inPredicate)
+ case in: InSubquery =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ // The join condition of a sub-query expression is a predicate, evaluated by the join that
+ // the sub-query is rewritten into.
+ case sq: SubqueryExpression =>
+ sq.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ case other =>
Review Comment:
Right, and the inconsistency with the `<=> false` test was real. The traversal now recognises the positions `ReplaceNullWithFalseInPredicate` rewrites for the same reason — the operand of an `EqualNullSafe` against TRUE, the predicate of an `If`, the conditions of a `CaseWhen` — as truth-only. `... OR ((c1 IN (SELECT col1 FROM t3n)) <=> true)` now returns `1`, as do the `IF` and `CASE WHEN` forms that `SimplifyConditionals` folds into it; all three are tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4274565628
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,412 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `attr` is produced by `outerPlan` and not by `subPlan`.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`.
+ * This matters when the two share ExprIds, which a self-join can leave for the optimizer to
+ * deduplicate (SPARK-21835): the EXISTS arms classify against the raw sub-query plan, while the
+ * IN arms classify against the plan that `dedupSubqueryOnSelfJoin` has already separated from
+ * the outer plan. Without this a single shared attribute would satisfy both sides of the
+ * classification, so a nested sub-query that references only the sub-query it is nested in
+ * would be taken for one referencing both plans and rejected. A nested sub-query resolves in
+ * the scope of that sub-query, so a shared attribute is one of its own.
+ */
+ private def referencedByOuterPlanOnly(
+ attr: Attribute,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): Boolean = {
+ outerPlan.outputSet.contains(attr) && !subPlan.outputSet.contains(attr)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
Review Comment:
Addressed on the current HEAD — apologies for the late reply.
Correct, the comment was wrong about one of the two collectors. `collectUnknownSensitiveInSubqueries` does open with
```scala
case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
```
so it prunes every subquery-free subtree. It is only the rejection collector that does not: that one is a plain `TreeNode.collect`, which visits every node. The comment now says exactly that, and that the guard buys the pruning collector only its top-level call
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4274204571
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +159,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftSemi, joinCond, subHint)
Project(p.output, join)
case (p, Not(Exists(sub, _, _, conditions, subHint))) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftAnti, joinCond, subHint)
Project(p.output, join)
case (p, InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val (joinCond, outerPlan) = rewriteExistentialExpr(inConditions ++ conditions, p)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(inConditions ++ conditions, p, dedupSub)
Review Comment:
Understood — I will not defer it. Taking it here.
To set expectations on what that adds to this PR, from the measurements in the thread above: the fix touches both statistics modes, so it changes estimates for every existing ExistenceJoin, including the ones produced by the pre-existing outer route that this PR does not touch. Eight approved TPC-DS plans contain an ExistenceJoin (q10, q35, q45 and their sf100 variants), so their golden files move too.
I will keep that in a separate commit on this branch — statistics change plus regenerated goldens — so the subquery-routing fix and the estimation change stay separately reviewable. Scope as you specified it: left-preserving row count, the appended Boolean in the output width, left-side-only attribute statistics, correct behaviour when the right row count is absent or zero, agreement between the CBO and size-only modes on the left-preserving invariant, and the build-side plan regression where an otherwise broadcastable left input stays broadcastable regardless of the right estimate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4286970341
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,540 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
Review Comment:
Correct, the "would hold either way" clause is stale from when both rejections raised the same condition. With an `EXISTS` outer predicate and this rejection removed, the join-condition handling raises `UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION`, which is not what this test asserts, so the assertion would fail rather than hold. I have kept `NOT IN` and corrected the explanation to say the query would still fail under `EXISTS` but on a different condition than the one asserted here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4066008156
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
Review Comment:
Thanks for pointing it out, just noted that one commit was missed, just pushed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4040695643
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -454,6 +470,70 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
}
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, so it is left in the join condition, where both plans are in scope. Such a
+ * sub-query is necessarily uncorrelated, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects. It is then either planned as
+ * an in-subquery filter or reported as unsupported by the rewrite of predicate sub-queries in
+ * join conditions.
Review Comment:
You are right, and this is worse than a doc error. I reproduced it exactly as you described.
With `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled=false`, on
`t1(a)=(1,2,3)`, `t2(c1)=(1,8,9)`, `t3(col1)=(3,9)`:
```sql
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2 WHERE a = c1 OR a IN (SELECT col1 FROM t3 WHERE col1 = c1));
```
| | result |
|---|---|
| PostgreSQL / DuckDB | `1` |
| master | `INTERNAL_ERROR` (attribute not found) |
| this PR, before the fixup | **`1, 3`** — silently wrong |
The plan shows why: the leftover keeps its correlation in the list query,
```
+- Join LeftSemi, ((col1#29 = c1#35) OR col1#29 IN (list#32 [c1#35 && (col1#36 = c1#35)]))
```
and `PlanSubqueries` then plans it as an uncorrelated `InSubqueryExec`, dropping `col1 = c1`. As you
noted, `InSubquery` has no `references` override, so `values :+ query` puts the outer reference in
`references` while the list query stays correlated to the inner query — exactly the shape my
"necessarily uncorrelated" sentence claimed could not exist. Turning an error into a wrong answer is
not an acceptable trade, so I took your suggestion and settled it in this PR rather than leaving it
to the doc:
* a leftover that is **correlated** now raises `0A000`
(`UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION`),
because its join condition cannot survive being planned in the join condition;
* a leftover that is **uncorrelated** has no join condition to lose and is still left in place.
The check lives in `rewriteExistentialExprInSubqueryPlan`, so it covers the `NOT IN` branch too, not
only the branches that go through `rewriteExistentialExprInJoinCondition`. The rejection is
deliberately independent of `decorrelatePredicateSubqueriesInJoinPredicate`: previously the shape
happened to be caught by the join-condition rewrite under the default confs and produced a wrong
answer with that conf off, which is why the bug was invisible by default. The new test asserts
`0A000` under **both** settings of that conf.
Correlation can only be to the sub-query plan here, since correlating to the outer plan as well
would need two levels of correlation and the Analyzer rejects that — the doc now says only that,
and no longer claims the leftover is uncorrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4139418052
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,206 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) && !effectivelyReferencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
Review Comment:
The rejection now reports the full Not node, and the regression asserts that subqueryExpression starts with NOT exactly for the NOT IN case.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4068594154","thread_id":"inline:4068594154","verdict_sha256":"15b8678aea32e2f6c0ae8bee1554ad3f1ae7e6de47363ee749ece337d2c0f9ac"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,206 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) && !effectivelyReferencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
Review Comment:
Generic NOT no longer propagates truthOnly; only direct NOT IN takes the separate null-aware path, which is now documented.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4068594158","thread_id":"inline:4068594158","verdict_sha256":"15b8678aea32e2f6c0ae8bee1554ad3f1ae7e6de47363ee749ece337d2c0f9ac"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,206 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) && !effectivelyReferencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ case Not(child) => unknownSensitiveInSubqueries(child, inPredicate)
+ case in: InSubquery =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ // The join condition of a sub-query expression is a predicate, evaluated by the join that
+ // the sub-query is rewritten into.
+ case sq: SubqueryExpression =>
+ sq.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ case other =>
+ other.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = false))
+ }
+
+ /**
+ * Returns true if `e` is a positive IN sub-query whose row comparison can evaluate to unknown,
+ * that is one that can return NULL rather than only TRUE or FALSE. An existence join cannot
+ * represent that third value, see [[rewriteExistentialExprInJoinCondition]]. NOT IN is excluded,
+ * as [[rewriteExistentialExprWithAttrs]] gives it a null-aware join condition of its own.
+ */
+ private def inSubqueryMayBeUnknown(e: Expression): Boolean = e match {
+ case InSubquery(values, listQuery) =>
+ values.zip(listQuery.plan.output).exists { case (v, o) => v.nullable || o.nullable }
Review Comment:
The suite now rejects a nullable compared value against a non-nullable, nonmatching list output under both legacy settings; removing v.nullable makes it fail.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4068594164","thread_id":"inline:4068594164","verdict_sha256":"15b8678aea32e2f6c0ae8bee1554ad3f1ae7e6de47363ee749ece337d2c0f9ac"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,372 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, the predicate of an `If` and the conditions of a `CaseWhen`.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) => Nil
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
+ unknownSensitiveInSubqueries(right, truthOnly)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
+ unknownSensitiveInSubqueries(right, truthOnly)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here. The
+ // same holds for the branch conditions of a conditional.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ unknownSensitiveInSubqueries(left, truthOnly = true)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ unknownSensitiveInSubqueries(right, truthOnly = true)
+ case If(predicate, trueValue, falseValue) =>
+ unknownSensitiveInSubqueries(predicate, truthOnly = true) ++
+ unknownSensitiveInSubqueries(trueValue, truthOnly = false) ++
Review Comment:
If and CaseWhen result branches now inherit the enclosing context, with non-foldable true/result/else cases verifying answer and routed placement.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4130355286","thread_id":"inline:4130355286","verdict_sha256":"15b8678aea32e2f6c0ae8bee1554ad3f1ae7e6de47363ee749ece337d2c0f9ac"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,372 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, the predicate of an `If` and the conditions of a `CaseWhen`.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) => Nil
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
+ unknownSensitiveInSubqueries(right, truthOnly)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
+ unknownSensitiveInSubqueries(right, truthOnly)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here. The
+ // same holds for the branch conditions of a conditional.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ unknownSensitiveInSubqueries(left, truthOnly = true)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ unknownSensitiveInSubqueries(right, truthOnly = true)
+ case If(predicate, trueValue, falseValue) =>
+ unknownSensitiveInSubqueries(predicate, truthOnly = true) ++
+ unknownSensitiveInSubqueries(trueValue, truthOnly = false) ++
+ unknownSensitiveInSubqueries(falseValue, truthOnly = false)
+ case CaseWhen(branches, elseValue) =>
+ branches.flatMap { case (condition, value) =>
+ unknownSensitiveInSubqueries(condition, truthOnly = true) ++
+ unknownSensitiveInSubqueries(value, truthOnly = false)
+ } ++ elseValue.toSeq.flatMap(unknownSensitiveInSubqueries(_, truthOnly = false))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) =>
+ collectOrDescend(not, in, truthOnly)
+ case Not(child) => unknownSensitiveInSubqueries(child, truthOnly = false)
+ case in: InSubquery =>
+ collectOrDescend(in, in, truthOnly)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.flatMap(unknownSensitiveInSubqueries(_, truthOnly = true)) ++
+ sq.getOuterAttrs.flatMap(unknownSensitiveInSubqueries(_, truthOnly = false))
+ case other =>
+ other.children.flatMap(unknownSensitiveInSubqueries(_, truthOnly = false))
+ }
+
+ /**
+ * Collects `reported` if the IN sub-query `in` would lose an observable unknown, and otherwise
+ * descends into the values it compares and into its list query. The descent matters because
+ * [[InSubquery]] is a [[Predicate]] rather than a [[SubqueryExpression]], so an IN sub-query
+ * nested in the values or in the hoisted join condition of another one is only reached from
+ * here.
+ */
+ private def collectOrDescend(
+ reported: Expression,
+ in: InSubquery,
+ truthOnly: Boolean): Seq[Expression] = {
+ if (!truthOnly && inSubqueryMayBeUnknown(in)) {
+ Seq(reported)
+ } else {
+ in.values.flatMap(unknownSensitiveInSubqueries(_, truthOnly = false)) ++
+ unknownSensitiveInSubqueries(in.query, truthOnly)
+ }
+ }
+
+ /**
+ * Returns true if the row comparison of the IN sub-query `in` can evaluate to unknown, that is
+ * if it can return NULL rather than only TRUE or FALSE. An existence join cannot represent that
+ * third value, see [[rewriteExistentialExprInJoinCondition]].
+ *
+ * This deliberately does not use `InSubquery.nullable`, which returns false for either side
Review Comment:
The Scaladoc now says legacy mode drops only list-output nullability, the typo is fixed, and nullable-left rejection is asserted with both flag values.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4130355295","thread_id":"inline:4130355295","verdict_sha256":"15b8678aea32e2f6c0ae8bee1554ad3f1ae7e6de47363ee749ece337d2c0f9ac"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5901153882
Thanks for the update. I confirmed that the PR description now names NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY and links SPARK-59862 for the pre-existing outer/projection three-valued-IN behavior.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"conversation:5891452714","thread_id":"conversation:5891452714","verdict_sha256":"15b8678aea32e2f6c0ae8bee1554ad3f1ae7e6de47363ee749ece337d2c0f9ac"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5884995716
I would keep full self-join preparation and the pre-existing dedupSubqueryOnSelfJoin asymmetry as a follow-up: current supported SQL inputs are analyzer-deduplicated, and the remaining conflict failure predates this PR. The effective-reference invariant is appropriate because it is derived from the compared values and hoisted conditions the rewrite actually consumes; I do not see a supported reused-ExprId counterexample on this head. The two error names are reasonable. Please file the released outer/projection three-valued-IN behavior as a separate JIRA. Also update the PR description's both-plan example to name NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"conversation:5853675261","thread_id":"conversation:5853675261","verdict_sha256":"2cd9f1ced1b3473e0a464ba4a11f3bf67e36b6ecf38ad62dfcd78c05b261f024"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6332638268
@cloud-fan I have addressed all the previous pending comments, Could you PTAL once you get time? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4427705961
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,822 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. The EXISTS form of this query fails on the same condition, raised by the same
+ // rejection in rewriteExistentialExprInSubqueryPlan before any join exists; were that
+ // rejection removed, the EXISTS form would instead reach the join-condition handling, which
+ // rejects it with UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION under the default
+ // configuration, so only the NOT IN form pins this rejection on its own.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: existential subquery carried by a rewritten nested subquery") {
+ // The nested IN over t4 is owned by the subquery plan, so it is rewritten into an existence
+ // join and the condition PullupCorrelatedPredicates hoisted into it is grafted onto that
+ // join. That condition carries a third-level IN correlated to t4, which references neither
+ // the outer nor the subquery plan and so is routed by neither of the guards above. It has to
+ // be rewritten against the inputs of the join just built: left in place, PlanSubqueries plans
+ // it as an in-subquery filter and drops its own correlated predicate col1 = k, which returns
+ // every row of t1 instead of the one row where the correlation holds.
+ withTempView("t1", "t2", "t4", "t5") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ // No k equals any c1, so only the third-level IN can satisfy the hoisted disjunction, and
+ // its correlated predicate col1 = k is false for every row of t4: dropping it makes the
+ // level-2 IN hold for c1 = 1 and returns 1, 2 and 3.
+ Seq((1, 100), (9, 7)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(1).toDF("col1").persist().createOrReplaceTempView("t5")
+ val query =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t4
+ | WHERE k = c1 OR col1 IN (SELECT col1 FROM t5 WHERE col1 = k)))""".stripMargin
+ // The routing must not depend on the rewrite of predicate subqueries in join conditions,
+ // which is what used to make these two configurations disagree: it reached the grafted
+ // condition only when enabled.
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswerAndPlan(query, Row(1) :: Nil)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: a carried subquery is classified as on the subquery-plan route") {
+ // The carried router applies the same rules as rewriteExistentialExprInSubqueryPlan, against
+ // the inputs of the existence join rather than the outer and the subquery plan.
+ withTempView("t1", "t2", "t4", "t5", "t3n", "t5n") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((9, 7)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(Some(3), None).toDF("x").persist().createOrReplaceTempView("t3n")
+ Seq((Some(3), 7)).toDF("x", "k").persist().createOrReplaceTempView("t5n")
+
+ // A carried NOT IN is collected as the Not over the IN, so the rewrite takes its null-aware
+ // arm. Taking the positive-IN arm instead leaves NOT exists over a condition that ignores
+ // NULL comparisons: 5 NOT IN (NULL) is unknown, so n4's (5, 100) row is excluded and the
+ // answer is empty, where the two-valued form returns every row of n1, that is 1 and 2.
+ withTempView("n1", "n2", "n4", "n5") {
+ Seq(1, 2).toDF("a").persist().createOrReplaceTempView("n1")
+ Seq(5).toDF("c1").persist().createOrReplaceTempView("n2")
+ Seq((5, 100)).toDF("col1", "k").persist().createOrReplaceTempView("n4")
+ Seq((Option.empty[Int], 100)).toDF("x", "y").persist().createOrReplaceTempView("n5")
+ val carriedNotIn =
+ """SELECT * FROM n1 WHERE EXISTS (
+ | SELECT 1 FROM n2 WHERE a = c1 OR c1 IN (
+ | SELECT col1 FROM n4
+ | WHERE k = c1 OR col1 NOT IN (SELECT x FROM n5 WHERE y = k)))""".stripMargin
+ Seq("true", "false").foreach { conf =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> conf) {
+ checkAnswer(sql(carriedNotIn), Nil)
+ }
+ }
+ }
+
+ // An uncorrelated carried IN is routed too, not left for the join-condition rewrite, which
+ // has no unknown guard and would make it a two-valued exists under the default
+ // configuration. Its unknown is observable here, so it is rejected under both settings.
+ val uncorrelatedUnknown =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (
+ | SELECT col1 FROM t4
+ | WHERE k = c1 OR ((col1 IN (SELECT x FROM t3n)) <=> false)))""".stripMargin
+ // A correlated one in the same position, which must report the same condition rather than
+ // the one about referencing both queries.
+ val correlatedUnknown =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (
+ | SELECT col1 FROM t4
+ | WHERE k = c1
+ | OR ((col1 IN (SELECT x FROM t5n WHERE x = k)) <=> false)))""".stripMargin
+ Seq(uncorrelatedUnknown, correlatedUnknown).foreach { query =>
+ Seq("true", "false").foreach { conf =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> conf) {
+ val e = intercept[AnalysisException](sql(query).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT", query)
+ }
+ }
+ }
+
+ // The same shape where only truth is observed is still rewritten and answers correctly.
+ Seq("true", "false").foreach { conf =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> conf) {
+ checkAnswer(
+ sql("""SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (
+ | SELECT col1 FROM t4
+ | WHERE k = c1 OR (col1 IN (SELECT x FROM t3n))))""".stripMargin),
+ Row(1) :: Nil)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: carried subqueries are routed on every route that builds an existence join") {
+ // rewriteCarriedExistentialExprs runs on every existence join this rule builds, not only on
+ // the ones the semi/anti join arms build for a top-level predicate subquery. On the routes
+ // below the outer EXISTS is itself rewritten into an existence join by
+ // rewriteExistentialExprWithAttrs, so the nested IN hoisted into its condition is carried
+ // rather than routed by the first-level classification, and only this router reaches it.
+ withTempView("t1", "t2", "t4", "t6") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ // The correlated nested IN is false for every c1 of t2, while the same IN without k = c1
+ // holds for c1 = 1, so losing that predicate turns every answer below into every row.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(1, 2, 3).toDF("b").persist().createOrReplaceTempView("t6")
+ val exists =
+ "EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t4 WHERE k = c1))"
+
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ // The Filter-predicate arm: under OR the EXISTS is not a conjunct of its own, so it
+ // goes through rewriteExistentialExpr rather than through the semi join arm.
+ checkAnswer(sql(s"SELECT * FROM t1 WHERE a > 100 OR $exists"), Row(1) :: Nil)
+ // handleUnaryNode: the EXISTS is a projection rather than a predicate.
+ checkAnswer(
+ sql(s"SELECT a, $exists AS e FROM t1"),
+ Row(1, true) :: Row(2, false) :: Row(3, false) :: Nil)
+ }
+ }
+
+ // The join-condition arm reaches the router only when it runs at all, which is what
+ // decorrelatePredicateSubqueriesInJoinPredicate gates; with it disabled the subquery stays
+ // in the join condition, which is pre-existing behaviour this PR does not change.
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "true") {
+ checkAnswer(
+ sql(s"SELECT t1.a FROM t1 JOIN t6 ON t1.a = t6.b AND $exists"), Row(1) :: Nil)
+ }
+ }
+ }
+
+ test("SPARK-59351: a carried IN shared between a bare and a negated position") {
+ // Filter pushdown substitutes the alias, so the same IN instance is carried bare and under
+ // a Not. Routing the bare one must not replace the one inside the Not, or the null-aware
+ // existence join built for the Not is left unreferenced and NOT exists over a two-valued
+ // match returns the rows where the IN is unknown. Standard SQL keeps only a = 1.
+ withTempView("s1", "s2", "s3") {
+ Seq(Some(1), Some(-1), None).toDF("a").persist().createOrReplaceTempView("s1")
+ Seq(1).toDF("c1").persist().createOrReplaceTempView("s2")
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("s3")
+ val query =
+ """SELECT s.a FROM (SELECT a, a IN (SELECT col1 FROM s3) AS b FROM s1) s
+ |WHERE s.a > 0 OR EXISTS (SELECT 1 FROM s2 WHERE s.a = s2.c1 OR s.b OR NOT s.b)"""
+ .stripMargin
+ Seq("true", "false").foreach { conf =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> conf) {
+ checkAnswer(sql(query), Row(1) :: Nil)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: a carried subquery referencing both inputs of its existence join") {
+ // The third-level IN takes its compared value c1 from the left input of the existence join
+ // built for the nested IN over t4, and its hoisted x = k reads the right one. Neither input
+ // can evaluate it, and left in place it is planned as an in-subquery filter that drops x = k.
+ withTempView("t1", "t2", "t4", "t5") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(1, 9).toDF("x").persist().createOrReplaceTempView("t5")
+ val query =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (
+ | SELECT col1 FROM t4
+ | WHERE k = c1 OR c1 IN (SELECT x FROM t5 WHERE x = k)))""".stripMargin
+ Seq("true", "false").foreach { conf =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> conf) {
+ val e = intercept[AnalysisException](sql(query).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY", conf)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: carried unknown-result rejection is limited to the subquery-plan route") {
+ // On the routes that predate this change the carried IN below was answered, and it is not
+ // rejected: its handling there is left to SPARK-59862. The columns are nullable but hold no
+ // NULL, so the two-valued and the three-valued answers agree and a rejection would be the
+ // only difference. The first query takes its compared value from the right input of the
+ // existence join built for the EXISTS, the second from the left one.
+ withTempView("t1", "t2", "t3") {
+ Seq(Some(1), Some(2), Some(3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(Some(1), Some(8), Some(9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq(Some(8)).toDF("col1").persist().createOrReplaceTempView("t3")
Review Comment:
Fixed in eaeb8f53e71. `t3` now holds a non-matching NULL. With each join-condition flag off, both the right-input and the left-input query must return only 1, keep the IN in the existence-join condition, and plan it through `InSubqueryExec`. Removing the in-place keep fails the case. Under the default the test only checks that the query isn't rejected, since the two-valued answer there is SPARK-59862.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub) via reviews" <re...@spark.apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6555580725
@cloud-fan Have addressed the above comments as well as some minor Nits i observed while addressing them. PTAL again. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4072209586
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -414,7 +433,8 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
existenceJoin, newCondition, subHint)
introducedAttrs += exists
exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
val exists = AttributeReference("exists", BooleanType, nullable = false)()
// Deduplicate conflicting attributes if any.
val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
Review Comment:
Confirmed, and the asymmetry is real. Building the state directly in a Catalyst test -- the analyzer's `DeduplicateRelations` removes it from every SQL query I could write, which is why no existing test covers it a nested sub-query that references only the sub-query it is nested in is reported as referencing both plans:
```
UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY
```
exactly as you predicted, because the single shared attribute satisfies both sides of the classification.
I did not move `dedupSubqueryOnSelfJoin` ahead of the routing. Doing that hits the risk in your own note -- "Moving deduplication without rewriting correlated condition references can create ambiguous or stale join expressions" -- and here it is not hypothetical. Deduplication aliases the sub-query plan's `a` to a new ExprId, while the hoisted condition still says `a`. The semi join's left side is the outer plan, which still produces `a`, so the condition would silently bind to the outer attribute instead of the sub-query's: a correlated predicate quietly turned into a self-reference, with no error. The IN arms avoid this only because they rebuild `inConditions` from
`dedupSub.output`; the hoisted `conditions` they pass through have the same exposure. So the classification is made collision-proof instead. `effectivelyReferencesPlanOnly` attributes an attribute that *both* plans produce to the sub-query plan, on the grounds that a nested sub-query resolves in the scope of the sub-query it sits in, so a shared ExprId is that sub-query's own. The routing then matches what the IN arms get from a deduplicated plan, without touching deduplication or the error it raises.
A second finding while testing this: in that state the rewrite cannot complete either way. `buildJoin` passes the semi join condition to `dedupSubqueryOnSelfJoin`, which has always rejected a condition naming a duplicated attribute, so the query ends at `_LEGACY_ERROR_TEMP_1212` (`conflictingAttributesInJoinConditionError`) -- the pre-existing behaviour you describe in the comment on the dedup anchor. A correlated EXISTS's hoisted condition always names sub-plan attributes, so any shared ExprId the condition mentions lands there. The routing fix therefore changes *which* error a user sees, from a new 0A000 that misdescribes the query to the long-standing conflict error that names the duplicated attribute, rather than making the query work. The new test in `RewriteSubquerySuite` pins that: it asserts the conflict error, and fails with `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY` if the classification is reverted. I have not tried to make that shape succeed. That needs dedupl
ication that rewrites the hoisted condition along with the plan, which is your "single preparation step" and changes a pre-existing error path for all four arms. Happy to take it on, here or separately, if you want it in scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114391450
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) && referencesPlan(sq, subPlan) &&
+ referencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => referencesPlan(sq, subPlan) && !referencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => referencesPlan(e, subPlan) && !referencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ case Not(child) => unknownSensitiveInSubqueries(child, inPredicate)
+ case in: InSubquery =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ // The join condition of a sub-query expression is a predicate, evaluated by the join that
+ // the sub-query is rewritten into.
+ case sq: SubqueryExpression =>
+ sq.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ case other =>
+ other.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = false))
+ }
+
+ /**
+ * Returns true if `e` is a positive IN sub-query whose row comparison can evaluate to unknown,
+ * that is one that can return NULL rather than only TRUE or FALSE. An existence join cannot
+ * represent that third value, see [[rewriteExistentialExprInJoinCondition]]. NOT IN is excluded,
+ * as [[rewriteExistentialExprWithAttrs]] gives it a null-aware join condition of its own.
+ */
+ private def inSubqueryMayBeUnknown(e: Expression): Boolean = e match {
Review Comment:
The divergence is deliberate and is now documented at the guard: what it protects is the
three-valued result the rewrite would drop, which does not depend on that flag, so reusing
`in.nullable` would make it unsound under it. Also switched to `listQuery.childOutputs`, which is the
intended accessor.
Your prediction about the flag checked out: under `spark.sql.legacy.inSubqueryNullability=true` the
folds take the IN out of the value position before the guard sees it, so two of the three predicates
stop reaching it. `<=> false` then returns the right answer anyway, because the fold produces a
`Not(InSubquery)` that keeps its null-aware rewrite; `IS NULL` is folded to false and returns `1`
where the answer is `1, 2, 3`. That last one is the flag's own non-conformance and is the same on
master, so I documented it rather than asserting it as expected, and the test asserts the default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114396708
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) && referencesPlan(sq, subPlan) &&
+ referencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => referencesPlan(sq, subPlan) && !referencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => referencesPlan(e, subPlan) && !referencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ case Not(child) => unknownSensitiveInSubqueries(child, inPredicate)
+ case in: InSubquery =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ // The join condition of a sub-query expression is a predicate, evaluated by the join that
+ // the sub-query is rewritten into.
+ case sq: SubqueryExpression =>
Review Comment:
Fixed. The `SubqueryExpression` arm now passes the truth-only flag for the hoisted join condition
only, via a new `getJoinCond` accessor next to the existing `getOuterAttrs`, and walks the outer
attributes as value positions, which is what the comment always said.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114400872
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
Review Comment:
Fixed with your guard, at the top of `rewriteExistentialExprInSubqueryPlan`, so a correlated EXISTS
or IN with no nested sub-query does one pattern test as before. `unknownSensitiveInSubqueries` also
now returns immediately for a sub-tree with neither pattern, which prunes sub-query-free branches of
an `OR` as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4056565047
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,100 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, so it is left in the join condition, where both plans are in scope. This is only
+ * correct for an uncorrelated sub-query: the join condition of a correlated one is dropped when
+ * it is planned as an in-subquery filter, which would silently change the result, so it is
+ * reported as unsupported instead. Note that such a sub-query can be correlated only to the
+ * sub-query plan, as being correlated to the outer plan as well would require two levels of
+ * correlation, which the Analyzer rejects.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A correlated sub-query referencing both plans has to stay in the join condition, where its
+ // own join condition is lost, so reject it rather than silently return a wrong result.
+ val unsupported = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) &&
+ referencesPlan(sq, subPlan) && referencesPlan(sq, outerPlan) => sq
+ })
+ if (unsupported.nonEmpty) {
+ throw QueryCompilationErrors.unsupportedCorrelatedSubqueryInJoinConditionError(unsupported)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
Review Comment:
Confirmed, and thank you — this was a wrong-result regression on the new route. Reproduced with
`t1(a)=(1,2,3)`, `t2(c1)=(1,8,9)`, `t3(col1)=(3,NULL)`, where the NULL matches nothing in `t2`:
```sql
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2 WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3)) <=> false));
```
| | result |
|---|---|
| PostgreSQL / DuckDB | `1` |
| master | `INTERNAL_ERROR` (attribute not found) |
| this PR, before this fixup | **`1, 2, 3`** |
and the mirror case `(c1 IN (SELECT col1 FROM t3)) IS NULL` returned `1` where it should return
`1, 2, 3`. Exactly as you describe: `NULL <=> false` is FALSE while `exists <=> false` is TRUE,
because the `exists` attribute is FALSE where IN is unknown.
One finding worth recording before the fix: **this is not specific to the new route.** The same
substitution on the outer route and in a projection is already wrong on released 4.0.1, with no
correlation involved:
```sql
-- 4.0.1 returns 1, 2 ; PostgreSQL/DuckDB return 1
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2 WHERE a = c1 OR ((a IN (SELECT col1 FROM t3)) <=> false));
-- 4.0.1 returns true, true, false ; PostgreSQL/DuckDB return false, false, false
SELECT a, ((a IN (SELECT col1 FROM t3)) <=> false) FROM t1;
```
So the owner-level representation you describe would fix a pre-existing bug as well as this one.
### What this fixup does
The distinction is only observable when the IN value escapes a predicate position. Within `AND`,
`OR` and `NOT` under a `Filter` or a join condition — which is what a hoisted correlated predicate
normally is — unknown and FALSE are indistinguishable, and the existing two-valued rewrite is
exact. So the fixup rejects precisely the observable case on the new route:
* the IN value reaches something other than a predicate **and** its row comparison can be unknown
(`values.zip(listQuery.plan.output).exists(_.nullable || _.nullable)`) -> rejected with
`0A000 ... NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT`, independent of
`decorrelatePredicateSubqueriesInJoinPredicate`;
* otherwise -> rewritten as before.
The ordinary shape therefore keeps working with nullable columns, which matters because real tables
are nullable and that is the case the JIRA is about:
```sql
-- returns 1, as PostgreSQL and DuckDB do, with t3 = (3, NULL)
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t3));
```
Leaving the subquery in the join condition is not an alternative here: under the default
configuration `RewritePredicateSubquery` re-offers the join it just produced to its own handling of
predicate subqueries in join conditions, which rebuilds the same two-valued existence join. That is
why the rejection is in the rewrite itself, and why the test asserts it under both settings of that
conf. NOT IN is rejected in that position too, since its null-aware join condition is equally
two-valued; its behaviour as a predicate is untouched.
### On the full representation change
I did not implement the TRUE/FALSE/NULL reconstruction in this PR. Reconstructing it needs two
existence joins per nested IN — one on the comparison and one on `cond OR isnull(cond)`, which is
what gives correct row-comparison semantics for multi-column IN, since `isnull` of the conjunction
is true exactly when no equality is FALSE and at least one is NULL — and the second join needs a
copy of the subquery plan with fresh expression ids. Applying that at the owner, as you ask, also
changes the outer and projection routes, i.e. long-standing plans and results, well beyond this
JIRA.
Given that, I would rather do it as a follow-up with its own JIRA and plan-stability review than
fold it in here, while this PR stops the new route from producing wrong rows. Happy to do it in
this PR instead if you prefer the whole scope in one change — and equally happy to file the
follow-up now with the 4.0.1 repros above, since the pre-existing half is a correctness bug in its
own right. Let me know which you want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4056572174
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,100 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, so it is left in the join condition, where both plans are in scope. This is only
+ * correct for an uncorrelated sub-query: the join condition of a correlated one is dropped when
+ * it is planned as an in-subquery filter, which would silently change the result, so it is
+ * reported as unsupported instead. Note that such a sub-query can be correlated only to the
+ * sub-query plan, as being correlated to the outer plan as well would require two levels of
+ * correlation, which the Analyzer rejects.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A correlated sub-query referencing both plans has to stay in the join condition, where its
+ // own join condition is lost, so reject it rather than silently return a wrong result.
+ val unsupported = conditions.flatMap(_.collect {
Review Comment:
Deliberate, and the comment now says so, close to your wording: a correlated subquery hidden under a
declined one would equally be planned without its join condition or reach execution unevaluable, so
it must be rejected even though nothing would have rewritten it. Agreed that narrowing the scan to
match the rewrite is the worse direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4274261907
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,476 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. The two plans share no ExprId in any
Review Comment:
You are right, and the comment was my mistake. I instrumented `planReferences` to print on any attribute present in both output sets and ran only `SubquerySuite`, which has no such shape, and then wrote the conclusion as if it covered the whole suite. Running the same instrumentation over `RewriteSubquerySuite` prints one hit, `COLLISION_SEEN a#44`, from the shared-attribute fixture.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4281442433
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,412 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
Review Comment:
Confirmed for the declined-parent case: carriesCorrelatedSubquery now rejects a both-plans parent that carries a correlated descendant, and carriedCorrelation asserts it under both settings. Rejecting the carrier instead of threading reachability state is fine for this boundary. The rewritten-parent boundary is a separate issue, discussed on r4257086397.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4248986904","thread_id":"inline:4248986904","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,375 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: nested IN subquery whose result can be unknown") {
+ withTempView("t1", "t2", "t2n", "t3n", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // A nullable compared value, against a subquery column that is not nullable.
+ Seq(Some(1), None).toDF("c1").persist().createOrReplaceTempView("t2n")
+ Seq((1, 1), (9, 9)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ // The NULL matches no value of t2, so c1 IN (SELECT col1 FROM t3n) is unknown there, and
+ // c1 NOT IN (SELECT col1 FROM t3n) is never true.
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // An existence join only records whether a row matched, so its exists attribute is false
+ // where IN is unknown. That is not observable while the value feeds a predicate, but it is
+ // once it reaches anything else, so these are rejected rather than rewritten. Both
+ // configurations are checked, as the rejection must not depend on the rewrite of predicate
+ // subqueries in join conditions.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> false",
+ "(c1 IN (SELECT col1 FROM t3n)) IS NULL",
+ // COALESCE to a non-false default does tell unknown from false.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), true)",
+ "(c1 NOT IN (SELECT col1 FROM t3n)) <=> false").foreach { predicate =>
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)")
+ .collect()
+ }
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT")
+ // A NOT IN must be reported as a NOT IN, not as the IN underneath it.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.startsWith("(NOT ") == predicate.contains("NOT IN"),
+ s"unexpected subqueryExpression for $predicate: $reported")
+ }
+ }
+ }
+
+ // The ordinary case: the value of the nested IN feeds a predicate, where unknown and
+ // false cannot be told apart, so it is still rewritten even though it can be unknown.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t3n))""".stripMargin,
+ Row(1) :: Nil)
+
+ // And a nested IN that cannot be unknown is rewritten even outside a predicate.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3)) <=> false))""".stripMargin,
+ Row(1) :: Row(2) :: Row(3) :: Nil)
+
+ // Only truth is observed by `<=> true`, by an IF predicate and by a CASE WHEN
+ // condition, so those are rewritten rather than rejected. SimplifyConditionals folds
+ // the latter two into the first, so ordinary SQL reaches this position.
+ //
+ // The result branches of a conditional are consumed wherever the conditional is, so
+ // under a filter predicate they are truth-only as well. The last two cases do not
+ // fold, so they exercise that inheritance rather than the rewrites above.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> true",
+ "IF(c1 IN (SELECT col1 FROM t3n), true, false)",
+ "CASE WHEN c1 IN (SELECT col1 FROM t3n) THEN true ELSE false END",
+ "IF(c1 > 0, c1 IN (SELECT col1 FROM t3n), false)",
+ "CASE WHEN c1 > 0 THEN c1 IN (SELECT col1 FROM t3n) ELSE false END",
+ "CASE WHEN c1 > 100 THEN false ELSE c1 IN (SELECT col1 FROM t3n) END",
+ // COALESCE to false maps unknown and false alike, so only truth is observed.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), false)")
+ .foreach { predicate =>
+ checkAnswerAndPlan(
+ s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)",
+ Row(1) :: Nil)
+ }
+
+ // The unknown can come from the compared value rather than from the subquery column.
+ // spark.sql.legacy.inSubqueryNullability makes InSubquery.nullable ignore the
+ // nullability of the list query columns, but it always keeps that of the compared
+ // values, and the guard re-derives both, so this case is rejected under either value.
+ Seq("true", "false").foreach { legacyNullability =>
+ withSQLConf(SQLConf.LEGACY_IN_SUBQUERY_NULLABILITY.key -> legacyNullability) {
+ val nullableValue = intercept[AnalysisException] {
+ sql("""SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2n
+ | WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3)) <=> false))""".stripMargin)
+ .collect()
+ }
+ assert(nullableValue.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT")
+ }
+ }
+
+ // An IN subquery nested in the values or in the hoisted condition of another one is
Review Comment:
Confirmed: nestedInValues now pins the values-side descent and asserts which IN is reported, and nestedInJoinCondition claims only the hoisted-condition descent.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4248986916","thread_id":"inline:4248986916","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,412 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `attr` is produced by `outerPlan` and not by `subPlan`.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`.
+ * This matters when the two share ExprIds, which a self-join can leave for the optimizer to
+ * deduplicate (SPARK-21835): the EXISTS arms classify against the raw sub-query plan, while the
+ * IN arms classify against the plan that `dedupSubqueryOnSelfJoin` has already separated from
+ * the outer plan. Without this a single shared attribute would satisfy both sides of the
+ * classification, so a nested sub-query that references only the sub-query it is nested in
+ * would be taken for one referencing both plans and rejected. A nested sub-query resolves in
+ * the scope of that sub-query, so a shared attribute is one of its own.
+ */
+ private def referencedByOuterPlanOnly(
+ attr: Attribute,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): Boolean = {
+ outerPlan.outputSet.contains(attr) && !subPlan.outputSet.contains(attr)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
Review Comment:
Confirmed: the comment now says only the rejection collect is unpruned, and that the early guard saves the pruning collector just its top-level call.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4248986922","thread_id":"inline:4248986922","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,412 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `attr` is produced by `outerPlan` and not by `subPlan`.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`.
+ * This matters when the two share ExprIds, which a self-join can leave for the optimizer to
+ * deduplicate (SPARK-21835): the EXISTS arms classify against the raw sub-query plan, while the
+ * IN arms classify against the plan that `dedupSubqueryOnSelfJoin` has already separated from
+ * the outer plan. Without this a single shared attribute would satisfy both sides of the
+ * classification, so a nested sub-query that references only the sub-query it is nested in
+ * would be taken for one referencing both plans and rejected. A nested sub-query resolves in
+ * the scope of that sub-query, so a shared attribute is one of its own.
+ */
+ private def referencedByOuterPlanOnly(
+ attr: Attribute,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): Boolean = {
+ outerPlan.outputSet.contains(attr) && !subPlan.outputSet.contains(attr)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
Review Comment:
Confirmed: effectiveReferences merges the child sets directly, and planReferences with the identity memo classifies each candidate once.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4248986929","thread_id":"inline:4248986929","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,476 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. The two plans share no ExprId in any
+ * shape reached by the suite: `DeduplicateRelations` has given the sub-query its own, and the
+ * EXISTS arms classify against the raw sub-query plan while the IN arms classify against the
+ * plan `dedupSubqueryOnSelfJoin` has already separated from the outer plan. The order is
+ * therefore defensive rather than something a query exercises, and it is the safe one: a
+ * nested sub-query resolves in the scope of the sub-query it is nested in, so an attribute
+ * both plans produce is one of that sub-query's own, and attributing it to `outerPlan` instead
+ * would let a single shared attribute satisfy both sides of the classification and have a
+ * nested sub-query referencing only the inner query rejected as referencing both.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite
+ * without rejecting the sub-query, as a sub-query referencing neither plan can be left in the
+ * join condition unharmed.
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
Review Comment:
Thanks for the careful repro, but f38599646be is not the pre-PR rule: it is this branch's merge of master and already contains rewriteExistentialExprInJoinCondition (`git diff --stat 6e5c61fb0a7 f38599646be` shows 129 insertions and 18 deletions in optimizer/subquery.scala). At the merge base 6e5c61fb0a7 the level-2 IN is rewritten on the outer plan with a condition that references t2.c1, so this query fails with INTERNAL_ERROR; the new route is what turns that into 1, 2, 3 when decorrelatePredicateSubqueriesInJoinPredicate is off. The same unvisited carried descendant also escapes the unknown-result guard under the default configuration. With t4(col1, k) = (9, 7) and t3n(x) = (3, NULL), `SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t4 WHERE k = c1 OR ((col1 IN (SELECT x FROM t3n)) <=> false)))` returns every row of t1 instead of 1: the level-3 IN reads only t4.col1, so `.filter(ownedBySubPlan)` drops it, and the join-condition handler th
en turns it into a two-valued exists. Please handle carried descendants in this PR. Your option (1), applying the same classification, guards and routing to the condition of the existence join built on the subquery plan, is the one I would prefer; rejecting existential descendants carried by a rewritten parent is an acceptable fallback, since these shapes fail with INTERNAL_ERROR at the merge base.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4257086397","thread_id":"inline:4257086397","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,476 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. The two plans share no ExprId in any
Review Comment:
Thanks, the reachability sentence now distinguishes analyzer plans from the fixture. One inconsistency remains in the same scaladoc. Its first paragraph says attributing a shared attribute to outerPlan would satisfy both sides and reject an inner-only subquery as referencing both, which is true only if the attribute counts for the outer plan as well. The second says the fixture ends in the same error whichever plan gets the attribute, which is true only for a swap: under the 'as well' reading, the fixture's nested IN (joinCond col1 = a) hits NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY before buildJoin's dedup check. Please pick one alternative and align both paragraphs and the fixture comment with it.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4257086406","thread_id":"inline:4257086406","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4133801137
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,372 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, the predicate of an `If` and the conditions of a `CaseWhen`.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) => Nil
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
+ unknownSensitiveInSubqueries(right, truthOnly)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
+ unknownSensitiveInSubqueries(right, truthOnly)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here. The
+ // same holds for the branch conditions of a conditional.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ unknownSensitiveInSubqueries(left, truthOnly = true)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ unknownSensitiveInSubqueries(right, truthOnly = true)
+ case If(predicate, trueValue, falseValue) =>
+ unknownSensitiveInSubqueries(predicate, truthOnly = true) ++
+ unknownSensitiveInSubqueries(trueValue, truthOnly = false) ++
Review Comment:
You are right, and this over-rejected. The branches of a conditional are consumed wherever the
conditional itself is, so under a filter predicate `IF(c1 > 0, c1 IN (...), false)` observes only
truth: whether the IN yields unknown or FALSE, the `If` yields unknown or FALSE and the filter drops
the row either way. Resetting the context at the branches made that a value position and rejected
it.
`If` now propagates the incoming context to its true and false values, and `CaseWhen` to its branch
values and its else value; the predicate of an `If` and the condition of a `CaseWhen` branch stay
truth-only, which is the part `ReplaceNullWithFalseInPredicate` also relies on.
Covered by three non-foldable cases, which exercise the inheritance rather than the folds:
`IF(c1 > 0, c1 IN (...), false)`, `CASE WHEN c1 > 0 THEN c1 IN (...) ELSE false END` and
`CASE WHEN c1 > 100 THEN false ELSE c1 IN (...) END`. All three return `1`, which is what PostgreSQL
and DuckDB return, and all three fail if the propagation is reverted. Rejection under a
value-sensitive parent such as `<=> false`, and the direct NOT IN behaviour, are unchanged and still
asserted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4257086397
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,476 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. The two plans share no ExprId in any
+ * shape reached by the suite: `DeduplicateRelations` has given the sub-query its own, and the
+ * EXISTS arms classify against the raw sub-query plan while the IN arms classify against the
+ * plan `dedupSubqueryOnSelfJoin` has already separated from the outer plan. The order is
+ * therefore defensive rather than something a query exercises, and it is the safe one: a
+ * nested sub-query resolves in the scope of the sub-query it is nested in, so an attribute
+ * both plans produce is one of that sub-query's own, and attributing it to `outerPlan` instead
+ * would let a single shared attribute satisfy both sides of the classification and have a
+ * nested sub-query referencing only the inner query rejected as referencing both.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite
+ * without rejecting the sub-query, as a sub-query referencing neither plan can be left in the
+ * join condition unharmed.
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
Review Comment:
**Blocking (P1):** This gate checks carried descendant correlation only for an existential that itself references both plans. A subplan-owned parent can therefore pass validation even when one of its retained condition carriers contains a deeper correlated ListQuery. The parent rewrite replaces that expression atomically without visiting the child, and with join-predicate subquery decorrelation disabled PlanSubqueries later ignores the child's nonempty joinCond, so distinguishing data can silently return extra rows.
**Verification:**
- **Regression:** Exercise a subplan-owned parent existential carrying a deeper correlated IN with distinguishing non-null data and verify structured rejection, rather than an extra row, with join-predicate decorrelation both enabled and disabled.
- **Behavior:** Cover neighboring uncorrelated and directly rewritable nested existential shapes to show that the added validation does not disable valid owner-based routing.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,476 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. The two plans share no ExprId in any
Review Comment:
**Nit (P3):** This comment says that no shape reached by the suite shares an ExprId, but the added RewriteSubquerySuite fixture deliberately gives the outer and subquery plans the same attribute, asserts their overlap, and runs Optimize. The rationale should distinguish analyzer-produced SQL plans from that optimizer-level fixture instead of describing the subplan-first branch as unexercised defensive code.
**Verification:**
- **Inspection:** Compare the ownership rationale with the shared-attribute fixture and confirm that their reachability and expected subplan-first classification statements agree.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,476 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. The two plans share no ExprId in any
+ * shape reached by the suite: `DeduplicateRelations` has given the sub-query its own, and the
+ * EXISTS arms classify against the raw sub-query plan while the IN arms classify against the
+ * plan `dedupSubqueryOnSelfJoin` has already separated from the outer plan. The order is
+ * therefore defensive rather than something a query exercises, and it is the safe one: a
+ * nested sub-query resolves in the scope of the sub-query it is nested in, so an attribute
+ * both plans produce is one of that sub-query's own, and attributing it to `outerPlan` instead
+ * would let a single shared attribute satisfy both sides of the classification and have a
+ * nested sub-query referencing only the inner query rejected as referencing both.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite
+ * without rejecting the sub-query, as a sub-query referencing neither plan can be left in the
+ * join condition unharmed.
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(sq)) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(ownedBySubPlan)
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) =
+ rewriteExistentialExprWithAttrs(conditions, subPlan, ownedBySubPlan)
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ // COALESCE(x, false) maps both unknown and FALSE to FALSE, so only truth is observed. A
+ // different default does not: COALESCE(x, true) is TRUE where x is unknown and FALSE where x
+ // is FALSE.
+ case Coalesce(child +: rest) if rest.headOption.contains(Literal.FalseLiteral) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) => collectOrDescend(not, in, truthOnly, reported)
+ case Not(child) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = false, reported)
+ case in: InSubquery => collectOrDescend(in, in, truthOnly, reported)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = true, reported))
+ sq.getOuterAttrs.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ case other =>
+ other.children.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ }
+
+ /**
+ * Appends `report` to `reported` if the IN sub-query `in` would lose an observable unknown, and
+ * otherwise descends into the values it compares and into its list query. The descent matters
+ * because [[InSubquery]] is a [[Predicate]] rather than a [[SubqueryExpression]], so an IN
+ * sub-query nested in the values or in the hoisted join condition of another one is only reached
+ * from here.
+ */
+ private def collectOrDescend(
+ report: Expression,
+ in: InSubquery,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = {
+ if (!truthOnly && inSubqueryMayBeUnknown(in)) {
+ reported += report
+ } else {
+ in.values.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ collectUnknownSensitiveInSubqueries(in.query, truthOnly, reported)
+ }
+ }
+
+ /**
+ * Returns true if the row comparison of the IN sub-query `in` can evaluate to unknown, that is
+ * if it can return NULL rather than only TRUE or FALSE. An existence join cannot represent that
+ * third value, see [[rewriteExistentialExprInJoinCondition]].
+ *
+ * This deliberately does not use `InSubquery.nullable`, which under
+ * `spark.sql.legacy.inSubqueryNullability` (SPARK-43413) reports only the nullability of the
+ * compared values and ignores that of the list query's columns. What is protected here is the
+ * three-valued result the rewrite would drop, which does not depend on that configuration, so
+ * the nullability is re-derived from both sides.
+ */
+ private def inSubqueryMayBeUnknown(in: InSubquery): Boolean = {
+ // Iterators, so a wide row-valued IN stops at the first nullable side rather than pairing up
+ // the whole arity first.
+ in.values.iterator.zip(in.query.childOutputs.iterator).exists {
Review Comment:
**Non-blocking (P2):** This guard scans every value/output pair, but the added UNKNOWN-sensitive regressions are all scalar. If the implementation accidentally inspected only the first pair, they would still pass while a nullable later compared value or list output bypassed rejection and collapsed UNKNOWN to FALSE. A row-valued case with a non-nullable first pair and nullability isolated to a later pair is needed to protect the all-fields contract.
**Verification:**
- **Regression:** Exercise row-valued nested IN with a non-nullable first pair and nullability isolated to a later compared value, then to a later list output, and verify structured UNKNOWN-result rejection in both cases.
- **Behavior:** Exercise the corresponding fully non-nullable row-valued shape and verify that owner-based rewriting remains supported.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala:
##########
@@ -161,4 +163,69 @@ class RewriteSubquerySuite extends PlanTest {
val existsExpected = LocalRelation(existsQuery.analyze.output).analyze
comparePlans(existsOptimized, existsExpected)
}
+
+ test("SPARK-59351: nested subquery of an EXISTS whose plan shares attributes with the outer " +
+ "plan") {
+ // The EXISTS arms route the nested subquery against the raw subquery plan, while the IN arms
+ // route it against the plan dedupSubqueryOnSelfJoin has already separated from the outer
+ // plan. A self-join can leave an ExprId shared between the two plans for the optimizer
+ // (SPARK-21835), and the analyzer is not involved here, so build that state directly: `a` is
+ // the same attribute in the outer and in the subquery plan.
+ val a = $"a".int
+ val outer = LocalRelation(a, $"b".int)
+ val sub = LocalRelation(a)
+ val nested = LocalRelation($"col1".int)
+ assert(outer.outputSet.intersect(sub.outputSet).nonEmpty,
+ "the two plans must share an attribute for this test to mean anything")
+
+ // The nested subquery references `a`, which only the subquery plan is meant to give it, and
+ // is correlated to it, so it must be rewritten against the subquery plan. Classifying `a` as
+ // a reference to the outer plan as well would report it as referencing both plans.
+ val nestedInSubquery = InSubquery(
+ Seq(a),
+ ListQuery(nested.select($"col1"), joinCond = Seq(EqualTo($"col1", a))))
+ val query = Filter(
+ Exists(sub, joinCond = Seq(Or(EqualTo($"b", a), nestedInSubquery))),
+ outer)
+
+ // The shared attribute is attributed to the subquery plan, so the nested subquery is routed
+ // there rather than reported as referencing both plans. What then reports this state is the
+ // conflict check that dedupSubqueryOnSelfJoin has always run over the semi join condition,
+ // which names the duplicated attribute: an attribute shared between the two plans that the
+ // hoisted condition mentions cannot be turned into a semi join whatever the routing does.
+ val e = intercept[AnalysisException](Optimize.execute(query))
+ assert(e.getCondition == "_LEGACY_ERROR_TEMP_1212",
+ s"expected the pre-existing conflict check to report this, got ${e.getCondition}")
+ }
+
+ test("SPARK-59351: the rewrite keeps the origin and the tags of the sub-query it replaces") {
+ // The rewrite of a nested existential sub-query builds its replacement by hand rather than
+ // through TreeNode.transformDownWithPruning, so it has to reproduce what that does: apply the
+ // rule under the origin of the node being replaced and copy its tags. Without this a runtime
+ // error raised from the rewritten condition quotes the enclosing operator instead of the
+ // sub-query fragment, which no result or plan assertion would notice.
+ val outer = LocalRelation($"a".int)
+ val inner = LocalRelation($"x".int)
+ val tag = TreeNodeTag[String]("SPARK-59351")
+ val origin = Origin(line = Some(42), startPosition = Some(7))
+ val a = outer.output.head
+ val x = inner.output.head
+ // Correlated, so that the rule's unary node handler picks the projection up at all.
+ val exists = CurrentOrigin.withOrigin(origin) {
+ Exists(inner, outerAttrs = Seq(a), joinCond = Seq(EqualTo(a, x)))
+ }
+ exists.setTagValue(tag, "kept")
+
+ // An Exists outside a Filter goes through the unary node handler, which is the path that
+ // builds the replacement by hand.
+ val rewritten = RewritePredicateSubquery(Project(Seq(Alias(exists, "e")()), outer))
+ val replacements = rewritten.expressions.flatMap(_.collect {
+ case attr: AttributeReference if attr.name == "exists" => attr
+ })
+ assert(replacements.size == 1, s"expected one exists attribute in\n$rewritten")
+ assert(replacements.head.origin == origin,
Review Comment:
**Non-blocking (P2):** This regression exercises only Exists, whose EqualTo was already constructed under the target origin, and it inspects only the projected replacement attribute. It would remain green if the positive-IN or NOT-IN arms assigned the enclosing operator's origin to their synthesized equality, null-aware wrapper, or combined condition, moving the SQLQueryContext reported by later failures. Those generated condition nodes need origin assertions as well.
**Verification:**
- **Regression:** Exercise positive IN and NOT IN replacements under a distinctive source origin and assert that each generated equality, null-aware wrapper, and combined condition used for diagnostics retains that origin.
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,426 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: nested IN subquery whose result can be unknown") {
+ withTempView("t1", "t2", "t2n", "t3n", "t3", "t4", "tb") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // A nullable compared value, against a subquery column that is not nullable.
+ Seq(Some(1), None).toDF("c1").persist().createOrReplaceTempView("t2n")
+ Seq((1, 1), (9, 9)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ // A boolean relation, to compare the result of a nested IN subquery against.
+ Seq(false).toDF("b").persist().createOrReplaceTempView("tb")
+ // The NULL matches no value of t2, so c1 IN (SELECT col1 FROM t3n) is unknown there, and
+ // c1 NOT IN (SELECT col1 FROM t3n) is never true.
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // An existence join only records whether a row matched, so its exists attribute is false
+ // where IN is unknown. That is not observable while the value feeds a predicate, but it is
+ // once it reaches anything else, so these are rejected rather than rewritten. Both
+ // configurations are checked, as the rejection must not depend on the rewrite of predicate
+ // subqueries in join conditions.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> false",
+ "(c1 IN (SELECT col1 FROM t3n)) IS NULL",
+ // COALESCE to a non-false default does tell unknown from false.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), true)",
+ "(c1 NOT IN (SELECT col1 FROM t3n)) <=> false").foreach { predicate =>
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)")
+ .collect()
+ }
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT")
+ // A NOT IN must be reported as a NOT IN, not as the IN underneath it.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.startsWith("(NOT ") == predicate.contains("NOT IN"),
+ s"unexpected subqueryExpression for $predicate: $reported")
+ }
+ }
+ }
+
+ // The ordinary case: the value of the nested IN feeds a predicate, where unknown and
+ // false cannot be told apart, so it is still rewritten even though it can be unknown.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t3n))""".stripMargin,
+ Row(1) :: Nil)
+
+ // And a nested IN that cannot be unknown is rewritten even outside a predicate.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3)) <=> false))""".stripMargin,
+ Row(1) :: Row(2) :: Row(3) :: Nil)
+
+ // Only truth is observed by `<=> true`, by an IF predicate and by a CASE WHEN
+ // condition, so those are rewritten rather than rejected. SimplifyConditionals folds
+ // the latter two into the first, so ordinary SQL reaches this position.
+ //
+ // The result branches of a conditional are consumed wherever the conditional is, so
+ // under a filter predicate they are truth-only as well. The last two cases do not
+ // fold, so they exercise that inheritance rather than the rewrites above.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> true",
+ "IF(c1 IN (SELECT col1 FROM t3n), true, false)",
+ "CASE WHEN c1 IN (SELECT col1 FROM t3n) THEN true ELSE false END",
+ "IF(c1 > 0, c1 IN (SELECT col1 FROM t3n), false)",
+ "CASE WHEN c1 > 0 THEN c1 IN (SELECT col1 FROM t3n) ELSE false END",
+ "CASE WHEN c1 > 100 THEN false ELSE c1 IN (SELECT col1 FROM t3n) END",
+ // COALESCE to false maps unknown and false alike, so only truth is observed.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), false)")
+ .foreach { predicate =>
+ checkAnswerAndPlan(
+ s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)",
+ Row(1) :: Nil)
+ }
+
+ // The unknown can come from the compared value rather than from the subquery column.
+ // spark.sql.legacy.inSubqueryNullability makes InSubquery.nullable ignore the
+ // nullability of the list query columns, but it always keeps that of the compared
+ // values, and the guard re-derives both, so this case is rejected under either value.
+ Seq("true", "false").foreach { legacyNullability =>
Review Comment:
**Non-blocking (P2):** The legacy-nullability loop uses nullable compared values, so both flag values still report them as nullable. The RHS-only-nullable cases run only under the default setting. A regression to InSubquery.nullable would therefore pass this suite even though legacy=true suppresses RHS-derived nullability and can let an observable UNKNOWN be rewritten as FALSE. Please cover a non-nullable compared value and nullable list output under both settings.
**Verification:**
- **Compatibility:** With a non-nullable compared value and nullable list output, verify the same UNKNOWN-result rejection and structured expression parameters under both legacy nullability settings.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +159,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftSemi, joinCond, subHint)
Project(p.output, join)
case (p, Not(Exists(sub, _, _, conditions, subHint))) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftAnti, joinCond, subHint)
Project(p.output, join)
case (p, InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val (joinCond, outerPlan) = rewriteExistentialExpr(inConditions ++ conditions, p)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(inConditions ++ conditions, p, dedupSub)
Review Comment:
The estimator predates this PR, but this change newly places it on the supported build side and can disable broadcast. Please fix both statistics modes and add the build-side plan regression here rather than defer it.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4139416473","thread_id":"inline:4139416473","verdict_sha256":"d615aea3940ba179c60ebaf5c4a82a1c974b3542a4ada38121d6365f9d19a83c"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6423267109
Thanks @cloud-fan, all findings from the 10-25 round are addressed in 2666748431d. NOT IN units are now routed before bare INs, which fixes the shared-IN repro (test added under both flag values). The scaladocs, the `carriedNotIn` test comment and the NOT IN arm comment are corrected, and I added a test for the carried both-inputs rejection.
On the decision question, I narrowed the unknown-result rejection to the sub-query-plan route and left the older routes to SPARK-59862, so no migration note is needed (test added). `SubquerySuite` passes in full and the PR description is updated. PTAL, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4427704532
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +417,649 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, a correlated IN or NOT IN is planned by `PlanSubqueries`
+ * as an in-subquery filter that drops its own hoisted condition, which returns extra rows, and
+ * a correlated EXISTS reaches execution as an unevaluable `Exists` and fails.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ *
+ * The unknown-result rejection applies only when `rejectsUnknownResult` is set, which is the
+ * case for the existence joins built on the sub-query-plan route, where the same shapes failed
+ * with an internal error. On the routes that predate this change an uncorrelated IN whose
+ * unknown result is observable is left where it is, as it was, and a correlated one is routed
+ * without the rejection. Their two-valued handling is tracked by SPARK-59862.
+ *
+ * A NOT IN is routed before a bare IN, since the substitution of a unit replaces every
+ * occurrence of it, including one inside a `Not` over the same instance.
+ */
+ private def rewriteCarriedExistentialExprs(
+ join: Join,
+ rejectsUnknownResult: Boolean): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
Review Comment:
Fixed in eaeb8f53e71. Each unit's left and right references are derived once from `effectiveReferences` into an `InputReferences`, memoised by identity as in `rewriteExistentialExprInSubqueryPlan`. The rejections, the keep decision and the substitution all read from it.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +417,649 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, a correlated IN or NOT IN is planned by `PlanSubqueries`
Review Comment:
Fixed in eaeb8f53e71. The scaladoc now says the hoisted condition is lost only with `decorrelatePredicateSubqueriesInJoinPredicate.enabled` off. Under the default, the `case j: Join` arm also visits the existence joins this rule builds, and the router adds independence from the setting plus the unknown-result guard. The "for a join the user wrote" sentence is gone. The carried-routes test comment says the same.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +417,649 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, a correlated IN or NOT IN is planned by `PlanSubqueries`
+ * as an in-subquery filter that drops its own hoisted condition, which returns extra rows, and
+ * a correlated EXISTS reaches execution as an unevaluable `Exists` and fails.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ *
+ * The unknown-result rejection applies only when `rejectsUnknownResult` is set, which is the
+ * case for the existence joins built on the sub-query-plan route, where the same shapes failed
+ * with an internal error. On the routes that predate this change an uncorrelated IN whose
+ * unknown result is observable is left where it is, as it was, and a correlated one is routed
+ * without the rejection. Their two-valued handling is tracked by SPARK-59862.
+ *
+ * A NOT IN is routed before a bare IN, since the substitution of a unit replaces every
+ * occurrence of it, including one inside a `Not` over the same instance.
+ */
+ private def rewriteCarriedExistentialExprs(
+ join: Join,
+ rejectsUnknownResult: Boolean): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothInputs.toSeq)
+ }
+ // Only the sub-query-plan route rejects. An existence join built on another route answered
+ // such a sub-query before this routing existed, three-valued where it was left in place, and
+ // that is left to SPARK-59862: see `rejectsUnknownResult`.
+ def uncorrelated(sq: Expression): Boolean =
+ !hasCorrelatedCondition(sq) && !carriesCorrelatedSubquery(negatedIn(sq))
+ if (rejectsUnknownResult) {
+ val unknownResult = carried.filter(sq => referencesOneInput(sq) && observesUnknown(sq))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult.toSeq)
+ }
+ }
+ // A NOT IN unit goes before a bare IN: the substitution of a unit replaces every occurrence
+ // of it in the condition, so a bare IN routed first would also replace the one inside a
+ // `Not` over the same instance, and the null-aware join built for that `Not` would be left
+ // unreferenced. The sort is stable, so the order within each kind is the order collected.
+ val routed = carried.sortBy {
+ case Not(_: InSubquery) => 0
+ case _ => 1
+ }.filterNot(sq => !rejectsUnknownResult && observesUnknown(sq) && uncorrelated(sq))
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ routed.filter(sq => referencesOneInput(sq) && isInput(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) =
+ rewriteExistentialExprWithAttrs(Seq(sq), p, rejectsUnknownResult = rejectsUnknownResult)
+ newCondition = newCondition.transform {
+ case e if e.fastEquals(sq) => rewritten.get
+ }
+ newInput
+ }
+ }
+ val newLeft = route(join.left, referencesInput(_, join.left))
+ val newRight = route(join.right, referencesInput(_, join.right))
+ if (newLeft.fastEquals(join.left) && newRight.fastEquals(join.right)) {
+ join
+ } else {
+ // Drop the `exists` attributes the nested existence joins introduced, keeping this join's
+ // own output, which already carries its own.
+ Project(join.output, join.copy(
+ left = newLeft, right = newRight, condition = Some(newCondition)))
+ }
+ }
+
+ /**
+ * Collects the existential sub-queries carried by the condition of an existence join, as the
+ * units [[rewriteExistentialExprWithAttrs]] rewrites: a NOT IN is collected as the [[Not]] over
+ * the [[InSubquery]], so the rewrite takes its null-aware arm and the unknown-result collector,
+ * which reports the same node, can be compared against it.
+ *
+ * A collected sub-query is not descended into. One nested below it is reached when it is
+ * rewritten, since the join built for it goes through the routing in turn, and one below a
+ * sub-query left in place is left with it, as everywhere else in this rule.
+ */
+ private def collectCarriedExistentialExprs(
+ expr: Expression,
+ collected: ArrayBuffer[Expression]): Unit = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return
+ }
+ expr match {
+ case sq @ Not(_: InSubquery) => collected += sq
+ case sq: Exists => collected += sq
+ case sq: InSubquery => collected += sq
+ case other => other.children.foreach(collectCarriedExistentialExprs(_, collected))
+ }
+ }
+
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true,
+ rejectsUnknownResult: Boolean = false): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan = rewriteCarriedExistentialExprs(
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint),
+ rejectsUnknownResult)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint), rejectsUnknownResult)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(
+ Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint), rejectsUnknownResult)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. A nested sub-query resolves in the scope
+ * of the sub-query it is nested in, so an attribute both plans produce is one of that
+ * sub-query's own.
+ *
+ * The probes are exclusive, so swapping their order would attribute such an attribute to
+ * `outerPlan` alone: `subPlanOnly` and `bothPlans` would both be false, and a nested sub-query
+ * whose only reference is the shared attribute would be declined and left in the join condition
+ * rather than routed to the sub-query plan that can evaluate it.
+ *
+ * No plan the analyzer produces from SQL reaches this: `DeduplicateRelations` has given the
+ * sub-query its own ExprIds, and the EXISTS arms classify against the raw sub-query plan while
+ * the IN arms classify against the plan `dedupSubqueryOnSelfJoin` has already separated from the
+ * outer plan. What does reach it is the shared-attribute fixture in `RewriteSubquerySuite`,
+ * which builds the overlap directly because the analyzer is not involved there. That fixture
+ * pins the shape, not the outcome of this ordering: under either order it ends in the same
+ * error, since `dedupSubqueryOnSelfJoin`'s pre-existing conflict check reports the duplicated
+ * attribute that the hoisted condition mentions.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite here.
+ * That does not leave the sub-query unexamined: one carried by a nested sub-query that is
+ * rewritten is routed against the inputs of the join built for it, see
+ * [[rewriteCarriedExistentialExprs]].
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * The rejection covers the sub-queries the sub-query plan owns. An IN sub-query that only
+ * references the outer plan is still rewritten two-valued, as it was before SPARK-59351, which
+ * SPARK-59862 tracks.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan,
+ keepsThreeValuedResiduals: Boolean = false): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(sq)) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above. Unless the caller keeps such
+ // a sub-query as a residual, in which case no existence join is built for it and there is
+ // nothing to reject: see `keepsThreeValuedResiduals`.
+ val unknownSensitive = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(ownedBySubPlan)
+ val (residual, unknownResult) = unknownSensitive.partition(sq =>
+ keepsThreeValuedResiduals && !hasCorrelatedCondition(sq) && !carriesCorrelatedSubquery(sq))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ def canRewrite(sq: Expression): Boolean =
+ ownedBySubPlan(sq) && !residual.exists(_.fastEquals(sq))
+ val (newCond, newSubPlan, _) =
+ rewriteExistentialExprWithAttrs(conditions, subPlan, canRewrite, rejectsUnknownResult = true)
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ // COALESCE(x, false) maps both unknown and FALSE to FALSE, so only truth is observed. A
+ // different default does not: COALESCE(x, true) is TRUE where x is unknown and FALSE where x
+ // is FALSE.
+ case Coalesce(child +: rest) if rest.headOption.contains(Literal.FalseLiteral) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query is handled as that IN is, by `collectOrDescend`: it is
+ // reported in a value position, where the unknown its null-aware rewrite preserves would be
Review Comment:
Fixed in eaeb8f53e71, with your wording: a NOT IN is reported in a value position, where the FALSE its null-aware rewrite yields in place of unknown would be observable.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -184,8 +188,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Use EXISTS if performance matters to you.
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ // The hoisted correlated predicates may carry existential sub-queries that only the
+ // sub-query plan can evaluate, see rewriteExistentialExprInJoinCondition.
+ // This arm returns a bare Join, which this rule does not offer to its own handling of
+ // predicate sub-queries in join conditions, so an uncorrelated sub-query left in the
+ // anti-join condition stays there and `InSubqueryExec` evaluates it with the
+ // three-valued semantics an existence join cannot carry. That is what this arm did
+ // before SPARK-59351, so it is not rejected here. It only holds while this NOT IN is
+ // the last sub-query conjunct of the Filter: the fold over the conjuncts wraps this
+ // anti join as `p` of a later EXISTS or IN arm, in `Project(p.output, Join(p, ...))`,
Review Comment:
Fixed in eaeb8f53e71, in both places. The arm comment and the residual test comment now say that any later sub-query conjunct wraps the anti join, in its own join or, for the generic arm, in a Filter over it. The arm comment also explains how the new residual choice depends on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4427702482
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -184,8 +188,20 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Use EXISTS if performance matters to you.
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ // The hoisted correlated predicates may carry existential sub-queries that only the
+ // sub-query plan can evaluate, see rewriteExistentialExprInJoinCondition.
+ // This arm returns a bare Join, which this rule does not offer to its own handling of
Review Comment:
Addressed together with r4414363550 in eaeb8f53e71.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothInputs.toSeq)
+ }
+ val unknownResult = carried.filter(sq => referencesOneInput(sq) && observesUnknown(sq))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult.toSeq)
+ }
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ carried.filter(sq => referencesOneInput(sq) && isInput(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) = rewriteExistentialExprWithAttrs(Seq(sq), p)
+ newCondition = newCondition.transform {
Review Comment:
You're right: routing NOT IN first was only equivalent while every collected unit was routed, and the keep policy for the pre-existing routes, added in the same commit, broke that. Fixed in eaeb8f53e71 as you suggested. `rewriteCarriedExistentialExprs` now substitutes by position: it walks the condition with the same matching as the collector, rewrites a routed unit where it was collected, and returns a kept unit whole. A routed bare IN can no longer reach the IN inside a kept `Not`. The ordering and its comments are removed.
To be upfront about the test: with the in-place policy from r4414363469 now in, a kept `Not(IN)` and a routed bare IN of the same instance no longer occur together in any configuration where the answer can tell them apart. With either flag off both are kept. Under the default the join-condition arm rewrites the kept one positively afterwards (SPARK-59862). So no SQL case fails on the substitution alone. I added your `(NOT s.b) = s2.f` repro to the shared-IN test with each flag off, where it returns only `a = 1`. That pins the combined behaviour, and the positional substitution keeps it from depending on the keep policy.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
Review Comment:
Addressed together with r4414363509 in eaeb8f53e71.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +417,649 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, a correlated IN or NOT IN is planned by `PlanSubqueries`
+ * as an in-subquery filter that drops its own hoisted condition, which returns extra rows, and
+ * a correlated EXISTS reaches execution as an unevaluable `Exists` and fails.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ *
+ * The unknown-result rejection applies only when `rejectsUnknownResult` is set, which is the
+ * case for the existence joins built on the sub-query-plan route, where the same shapes failed
+ * with an internal error. On the routes that predate this change an uncorrelated IN whose
+ * unknown result is observable is left where it is, as it was, and a correlated one is routed
+ * without the rejection. Their two-valued handling is tracked by SPARK-59862.
+ *
+ * A NOT IN is routed before a bare IN, since the substitution of a unit replaces every
+ * occurrence of it, including one inside a `Not` over the same instance.
+ */
+ private def rewriteCarriedExistentialExprs(
+ join: Join,
+ rejectsUnknownResult: Boolean): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothInputs.toSeq)
+ }
+ // Only the sub-query-plan route rejects. An existence join built on another route answered
+ // such a sub-query before this routing existed, three-valued where it was left in place, and
+ // that is left to SPARK-59862: see `rejectsUnknownResult`.
+ def uncorrelated(sq: Expression): Boolean =
+ !hasCorrelatedCondition(sq) && !carriesCorrelatedSubquery(negatedIn(sq))
+ if (rejectsUnknownResult) {
+ val unknownResult = carried.filter(sq => referencesOneInput(sq) && observesUnknown(sq))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult.toSeq)
+ }
+ }
+ // A NOT IN unit goes before a bare IN: the substitution of a unit replaces every occurrence
+ // of it in the condition, so a bare IN routed first would also replace the one inside a
+ // `Not` over the same instance, and the null-aware join built for that `Not` would be left
+ // unreferenced. The sort is stable, so the order within each kind is the order collected.
+ val routed = carried.sortBy {
+ case Not(_: InSubquery) => 0
+ case _ => 1
+ }.filterNot(sq => !rejectsUnknownResult && observesUnknown(sq) && uncorrelated(sq))
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ routed.filter(sq => referencesOneInput(sq) && isInput(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) =
+ rewriteExistentialExprWithAttrs(Seq(sq), p, rejectsUnknownResult = rejectsUnknownResult)
+ newCondition = newCondition.transform {
+ case e if e.fastEquals(sq) => rewritten.get
+ }
+ newInput
+ }
+ }
+ val newLeft = route(join.left, referencesInput(_, join.left))
+ val newRight = route(join.right, referencesInput(_, join.right))
+ if (newLeft.fastEquals(join.left) && newRight.fastEquals(join.right)) {
+ join
+ } else {
+ // Drop the `exists` attributes the nested existence joins introduced, keeping this join's
+ // own output, which already carries its own.
+ Project(join.output, join.copy(
+ left = newLeft, right = newRight, condition = Some(newCondition)))
+ }
+ }
+
+ /**
+ * Collects the existential sub-queries carried by the condition of an existence join, as the
+ * units [[rewriteExistentialExprWithAttrs]] rewrites: a NOT IN is collected as the [[Not]] over
+ * the [[InSubquery]], so the rewrite takes its null-aware arm and the unknown-result collector,
+ * which reports the same node, can be compared against it.
+ *
+ * A collected sub-query is not descended into. One nested below it is reached when it is
+ * rewritten, since the join built for it goes through the routing in turn, and one below a
+ * sub-query left in place is left with it, as everywhere else in this rule.
+ */
+ private def collectCarriedExistentialExprs(
+ expr: Expression,
+ collected: ArrayBuffer[Expression]): Unit = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return
+ }
+ expr match {
+ case sq @ Not(_: InSubquery) => collected += sq
+ case sq: Exists => collected += sq
+ case sq: InSubquery => collected += sq
+ case other => other.children.foreach(collectCarriedExistentialExprs(_, collected))
+ }
+ }
+
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true,
+ rejectsUnknownResult: Boolean = false): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan = rewriteCarriedExistentialExprs(
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint),
+ rejectsUnknownResult)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint), rejectsUnknownResult)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(
+ Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint), rejectsUnknownResult)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. A nested sub-query resolves in the scope
+ * of the sub-query it is nested in, so an attribute both plans produce is one of that
+ * sub-query's own.
+ *
+ * The probes are exclusive, so swapping their order would attribute such an attribute to
+ * `outerPlan` alone: `subPlanOnly` and `bothPlans` would both be false, and a nested sub-query
+ * whose only reference is the shared attribute would be declined and left in the join condition
+ * rather than routed to the sub-query plan that can evaluate it.
+ *
+ * No plan the analyzer produces from SQL reaches this: `DeduplicateRelations` has given the
+ * sub-query its own ExprIds, and the EXISTS arms classify against the raw sub-query plan while
+ * the IN arms classify against the plan `dedupSubqueryOnSelfJoin` has already separated from the
+ * outer plan. What does reach it is the shared-attribute fixture in `RewriteSubquerySuite`,
+ * which builds the overlap directly because the analyzer is not involved there. That fixture
+ * pins the shape, not the outcome of this ordering: under either order it ends in the same
+ * error, since `dedupSubqueryOnSelfJoin`'s pre-existing conflict check reports the duplicated
+ * attribute that the hoisted condition mentions.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite here.
+ * That does not leave the sub-query unexamined: one carried by a nested sub-query that is
+ * rewritten is routed against the inputs of the join built for it, see
+ * [[rewriteCarriedExistentialExprs]].
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * The rejection covers the sub-queries the sub-query plan owns. An IN sub-query that only
+ * references the outer plan is still rewritten two-valued, as it was before SPARK-59351, which
+ * SPARK-59862 tracks.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan,
+ keepsThreeValuedResiduals: Boolean = false): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(sq)) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above. Unless the caller keeps such
+ // a sub-query as a residual, in which case no existence join is built for it and there is
+ // nothing to reject: see `keepsThreeValuedResiduals`.
+ val unknownSensitive = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(ownedBySubPlan)
+ val (residual, unknownResult) = unknownSensitive.partition(sq =>
Review Comment:
Agreed, and implemented as you recommended in eaeb8f53e71.
* **NOT IN arm:** the Filter fold now tracks each conjunct's position. When nothing reaches the anti join later, the arm keeps every uncorrelated IN or NOT IN that the sub-query plan owns as a residual, truth-only ones included, as at the merge base. That is when the NOT IN is the last sub-query conjunct, or when either `decorrelatePredicateSubqueriesInJoinPredicate.enabled` or `optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is off. Otherwise today's handling stays: a truth-only nested NOT IN is rewritten null-aware, so the `case j: Join` arm doesn't rewrite its IN positively. `keepsThreeValuedResiduals` is replaced by a three-state `ThreeValuedResiduals`.
* **Carried router:** on the routes that don't reject, an uncorrelated carried IN or NOT IN is also left in place when either flag is off.
* Both read the flags through one helper, `joinConditionRewriteReachesUncorrelatedIn`, next to a comment in the `case j: Join` arm saying the two must stay aligned. Correlated units, units carrying a correlated sub-query, EXISTS, and the sub-query-plan route are unchanged.
Tests:
* The NOT IN residual test asserts that both the truth-only nested IN and your nested NOT IN case are kept in place: no existence join in the executed plan, and an `InSubqueryExec`.
* A new case, `... AND EXISTS (SELECT 1 FROM t2 WHERE c1 >= a)` with a NULL in t3n, returns the standard answer under all three flag settings.
* An uncorrelated carried NOT IN stays in its existence-join condition with each flag off.
I checked that the tests discriminate: forcing the old residual policy, dropping the carried keep with flags off, or keeping all residuals when a later conjunct follows each fails a case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5620858141
+ @viirya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5771487320
Thanks for the update. The previous effective-reference and configuration-name fixes are present. On the current head I still found two nullable-IN gaps in the new inner-plan route: the safety walk stops at a predicate-position InSubquery without inspecting its join condition, and the legacy IN-nullability mode can normalize an observable `<=> false` use into a form that bypasses the guard. I have included both concrete paths in the review.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"conversation:5764348512","thread_id":"conversation:5764348512","verdict_sha256":"5e51400791b44cf792d02bbc59d2837419644768a5d9aa2d5eb0b180f6ad74e1"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4072311118
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +158,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
Review Comment:
Accepted in full — this was a regression I introduced when I replaced the traversal, not a
pre-existing gap, and your reading of the two lines is exactly right.
Each arm now builds its replacement through a small helper,
```scala
def replacing(sq: Expression)(replacement: => Expression): Expression = {
val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
newExpr.copyTagsFrom(sq)
newExpr
}
```
so everything an arm constructs — the `exists` attribute, the join conditions and the `Join` itself
— is built under the origin of the subquery expression being replaced, and that expression's tags
are carried over, which is what `transformDownWithPruning` did.
Your note about where the cost lands is what decided the shape of the fix: `handleUnaryNode` sends
every `UnaryNode` through this method with the default `canRewrite`, so the repair had to sit in the
arms themselves rather than in the new routing callers. The pruning check and the unchanged-node
fast path are unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5919025977
Thanks for your comments @cloud-fan I have addressed all the nits, while the comment: https://github.com/apache/spark/pull/58656#discussion_r4139416473 is pre-exiting, happy to take that up as a seperate follow-up PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114432613
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) && referencesPlan(sq, subPlan) &&
+ referencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => referencesPlan(sq, subPlan) && !referencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => referencesPlan(e, subPlan) && !referencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
Review Comment:
All three points were right, and all three are fixed.
**The error named the wrong operator.** The rejection now reports the `Not` node rather than the
`InSubquery` underneath it, so `Not.sql` renders the user's NOT IN. While fixing it the message
template read wrongly for the negated form, so it no longer begins "The IN subquery": it is now
"The subquery `<subqueryExpression>` is nested in a correlated condition of another subquery, and
its result can be unknown where it is not only tested for being true." The tests now assert
`subqueryExpression` as well as the condition, including that it starts with `(NOT ` exactly for the
NOT IN predicate, which is the assertion that was missing.
**The arm is no longer redundant.** It is what supplies the `Not` node to the diagnostic, so
deleting it now changes behaviour, and it no longer duplicates a condition that could drift: both
arms call one helper that either reports the node or descends into the sub-query's values and list
query.
**The scaladoc contradiction is gone.** "NOT IN is excluded" was wrong — NOT IN is rejected in a
value position, for the same reason as IN, since its null-aware rewrite is two-valued as well. The
scaladoc on `inSubqueryMayBeUnknown` now says only what the function does, and the contract about
NOT lives where the traversal decides it, with the explanation that a NOT directly over an IN is
safe because its null-aware join condition is TRUE where the comparison is unknown, so the negated
`exists` is FALSE there.
Your comment also prompted two related fixes in the same traversal, from your other notes: it now
descends into an IN sub-query it does not collect, so a nested one in the values or in the hoisted
condition is no longer skipped, and `<=> true`, `IF` and `CASE WHEN` conditions are recognised as
truth-only rather than rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5748836825
Thank you @LuciferYang and @cloud-fan for the comments, have made the required changes to address them PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4072209586
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -414,7 +433,8 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
existenceJoin, newCondition, subHint)
introducedAttrs += exists
exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
val exists = AttributeReference("exists", BooleanType, nullable = false)()
// Deduplicate conflicting attributes if any.
val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
Review Comment:
Taking the three parts separately.
**(b) is confirmed**, and it is what the new `RewriteSubquerySuite` test ends up asserting. With the
outer and the subquery plan sharing an attribute, the EXISTS arm reaches `dedupSubqueryOnSelfJoin`
through `buildJoin` with the condition, and the query fails with `_LEGACY_ERROR_TEMP_1212`:
```
Found conflicting attributes a#44 in the condition joining outer plan:
LocalRelation [a#44, b#45]
and subplan:
'Join ExistenceJoin(exists#49), ((a#44 = 'col1) AND ('col1 = a#44))
```
As you say, a correlated EXISTS's hoisted condition always names subplan attributes, so a collision
the condition mentions always lands there. That also bounds what the routing fix can achieve: it
decides which error appears, not whether the query runs.
**(a) I could not reproduce.** Your `c1 IN (SELECT a FROM t1)` example comes out of the analyzer
already deduplicated, and with plan change validation on, that query and the other self-join shapes
I tried — EXISTS, NOT EXISTS and IN over the same relation, with a nested IN and with a nested
EXISTS — all give plans with no missing input, `duplicateResolved` joins and `resolved = true`, and
the answers PostgreSQL and DuckDB give. So I have no `checkIfSameExprIdNotReused` failure to fix,
and I am not treating the concern as unfounded: it is a structural observation about where the
anchor now sits, and the reason it does not bite may simply be that
`dedupSubqueryOnSelfJoin` rejects the colliding shapes first, per (b). If you have a query that
reaches the optimizer with the conflict intact, I will add it as a test and fix what it shows.
**On the missing coverage**, "there is no test here where the nested relation overlaps either plan"
is now addressed: the new test in `RewriteSubquerySuite` builds that overlap directly, since the
analyzer will not hand it to us, and pins both the classification and the error it ends at.
**On the asymmetry inside the helper** — called with the condition from `buildJoin` and so able to
throw, called without one from the IN arms and so only aliasing — I have left it as it is. It is
pre-existing, and changing it is the same work as making the colliding EXISTS shape succeed, which
belongs with a preparation step across all four arms rather than in this JIRA. Happy to do it here
if you would rather it were not deferred.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4072209586
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -414,7 +433,8 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
existenceJoin, newCondition, subHint)
introducedAttrs += exists
exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
val exists = AttributeReference("exists", BooleanType, nullable = false)()
// Deduplicate conflicting attributes if any.
val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
Review Comment:
Taking the three parts separately.
**(b) is confirmed**, and it is what the new `RewriteSubquerySuite` test ends up asserting. With the
outer and the subquery plan sharing an attribute, the EXISTS arm reaches `dedupSubqueryOnSelfJoin`
through `buildJoin` with the condition, and the query fails with `_LEGACY_ERROR_TEMP_1212`:
```
Found conflicting attributes a#44 in the condition joining outer plan:
LocalRelation [a#44, b#45]
and subplan:
'Join ExistenceJoin(exists#49), ((a#44 = 'col1) AND ('col1 = a#44))
```
As you say, a correlated EXISTS's hoisted condition always names subplan attributes, so a collision
the condition mentions always lands there. That also bounds what the routing fix can achieve: it
decides which error appears, not whether the query runs.
**(a) I could not reproduce.** Your `c1 IN (SELECT a FROM t1)` example comes out of the analyzer
already deduplicated, and with plan change validation on, that query and the other self-join shapes
I tried — EXISTS, NOT EXISTS and IN over the same relation, with a nested IN and with a nested
EXISTS — all give plans with no missing input, `duplicateResolved` joins and `resolved = true`, and
the answers PostgreSQL and DuckDB give. So I have no `checkIfSameExprIdNotReused` failure to fix,
and I am not treating the concern as unfounded: it is a structural observation about where the
anchor now sits, and the reason it does not bite may simply be that
`dedupSubqueryOnSelfJoin` rejects the colliding shapes first, per (b). If you have a query that
reaches the optimizer with the conflict intact, I will add it as a test and fix what it shows.
**On the missing coverage**, "there is no test here where the nested relation overlaps either plan"
is now addressed: the new test in `RewriteSubquerySuite` builds that overlap directly, since the
analyzer will not hand it to us, and pins both the classification and the error it ends at.
**On the asymmetry inside the helper** — called with the condition from `buildJoin` and so able to
throw, called without one from the IN arms and so only aliasing — I have left it as it is. It is
pre-existing, and changing it is the same work as making the colliding EXISTS shape succeed, which
belongs with a preparation step across all four arms rather than in this JIRA. Happy to do it here
if you would rather it were not deferred.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4072358573
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
Review Comment:
Fixed, and taken one step further than suggested. Swapping the operands to
`plan.outputSet.intersect(e.references).nonEmpty` does iterate the small side, but it still
materialises a `LinkedHashSet` from the references for a boolean that is discarded, so the helper
now does
```scala
e.references.exists(plan.outputSet.contains)
```
which walks the references and probes the output set, allocating nothing at all. The same change is
applied to `effectivelyReferencesPlan` and `effectivelyReferencesPlanOnly`, which are on the same
path, and as you note the `case j: Join` branch picks it up through the shared helper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6183220982
Thanks for the extensive comments @cloud-fan - Have addressed all the previous comments, also have addressed https://github.com/apache/spark/pull/58656#discussion_r4139416473 as part of same PR, Could you PTAL again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6126017806
Thanks for the ping. I re-reviewed the current head and will include the remaining correctness and planning issues in the review.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"conversation:6098592689","thread_id":"conversation:6098592689","verdict_sha256":"0352a447f186aabca77a56585861c6bf45cf7f50c6a97298b59403fd37e21eb3"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4346076647
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,568 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is deliberately left where it is. It has no hoisted condition to
+ * lose, and `InSubqueryExec` evaluates it with the three-valued semantics that the `exists`
+ * attribute of an existence join cannot carry.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ // Only a correlated sub-query has to be consumed here, see above.
+ val carried = condition.collect {
+ case sq @ (_: Exists | _: InSubquery) if hasCorrelatedCondition(sq) => sq
+ }
+ if (carried.isEmpty) {
+ return join
+ }
+ // Where the value of a carried IN is observed beyond being TRUE, an existence join cannot
+ // stand in for it, exactly as on the routes above.
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+
+ val rejected = carried.filter { sq =>
+ observesUnknown(sq) ||
+ (referencesInput(sq, join.left) && referencesInput(sq, join.right))
+ }
+ if (rejected.nonEmpty) {
+ // A sub-query that needs both inputs can be rewritten against neither, and one whose
+ // unknown is observable cannot become an `exists` attribute at all.
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(rejected)
+ }
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ carried.filter(sq => isInput(sq) && !rejected.contains(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) = rewriteExistentialExprWithAttrs(Seq(sq), p)
Review Comment:
**Blocking (P1):** `carried` holds the `InSubquery` under a `NOT IN`, not the `Not`, so this call rewrites it through the positive-IN arm (`value = output AND hoisted`) instead of the null-aware `Not(InSubquery)` arm, and the `transform` below replaces only the IN, leaving `NOT exists` over a condition that ignores NULL comparisons. For example, with t1(a) = 1, 2; t2(c1) = 5; t4(col1, k) = (5, 100); t5(x, y) = (NULL, 100):
```sql
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (
SELECT col1 FROM t4 WHERE k = c1 OR col1 NOT IN (SELECT x FROM t5 WHERE y = k)))
```
`5 NOT IN (NULL)` is unknown, so standard SQL excludes the t4 row and returns no rows. Here `5 = NULL` never matches, `exists` is FALSE, `NOT exists` is TRUE, and both rows of t1 come back. At the merge base this shape failed with INTERNAL_ERROR.
The same unit mismatch affects the rejection: `unknownSensitiveInSubqueries` reports the `Not` node for a NOT IN, but `observesUnknown` compares it with the bare `InSubquery` in `carried`, so a carried NOT IN in a value position such as `(col1 NOT IN (...)) <=> false` is routed instead of rejected.
See **Shared repair plan 1** in the review body.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,568 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is deliberately left where it is. It has no hoisted condition to
+ * lose, and `InSubqueryExec` evaluates it with the three-valued semantics that the `exists`
+ * attribute of an existence join cannot carry.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ // Only a correlated sub-query has to be consumed here, see above.
+ val carried = condition.collect {
+ case sq @ (_: Exists | _: InSubquery) if hasCorrelatedCondition(sq) => sq
+ }
+ if (carried.isEmpty) {
+ return join
+ }
+ // Where the value of a carried IN is observed beyond being TRUE, an existence join cannot
+ // stand in for it, exactly as on the routes above.
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+
+ val rejected = carried.filter { sq =>
+ observesUnknown(sq) ||
+ (referencesInput(sq, join.left) && referencesInput(sq, join.right))
+ }
+ if (rejected.nonEmpty) {
+ // A sub-query that needs both inputs can be rewritten against neither, and one whose
+ // unknown is observable cannot become an `exists` attribute at all.
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(rejected)
Review Comment:
**Non-blocking (P2):** `rejected` mixes the two rejection reasons, but this always raises `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY`, whose message says the sub-query references both the outer query and the sub-query it is nested in. A carried IN rejected only because its unknown result is observable references neither, so users and tools matching `getCondition` get the wrong sub-condition and a false explanation. For example, with a nullable `t5n.x`:
```sql
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (
SELECT col1 FROM t4 WHERE k = c1 OR ((col1 IN (SELECT x FROM t5n WHERE x = k)) <=> false)))
```
The level-3 IN references only t4, so the first-level guard skips it; it is carried by the existence join on t2 and rejected here. `a > 5 OR EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t4n WHERE k = c1)) <=> false))` reaches the same throw through the Filter-predicate arm, while the same disjunct under a top-level EXISTS raises `NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT`, as SubquerySuite asserts. No test reaches this throw.
See **Shared repair plan 1** in the review body.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,568 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is deliberately left where it is. It has no hoisted condition to
+ * lose, and `InSubqueryExec` evaluates it with the three-valued semantics that the `exists`
+ * attribute of an existence join cannot carry.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ // Only a correlated sub-query has to be consumed here, see above.
+ val carried = condition.collect {
+ case sq @ (_: Exists | _: InSubquery) if hasCorrelatedCondition(sq) => sq
Review Comment:
**Blocking (P1):** Only carried sub-queries with a non-empty hoisted condition are collected here, so an uncorrelated one stays in the grafted condition. The scaladoc above assumes `InSubqueryExec` then evaluates it three-valued, but with `decorrelatePredicateSubqueriesInJoinPredicate.enabled` and `optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` at their default `true`, the top-down pass reaches this existence join in the `case j: Join` arm, which rewrites the IN against the input it references into a two-valued `exists`. The first-level unknown guard did not catch it either, because the sub-query plan does not own it. This is the second case from r4281444233, with t1(a) = 1, 2, 3; t2(c1) = 1, 8, 9; t4(col1, k) = (9, 7); t3n(x) = 3, NULL:
```sql
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (
SELECT col1 FROM t4 WHERE k = c1 OR ((col1 IN (SELECT x FROM t3n)) <=> false)))
```
`9 IN (3, NULL)` is unknown, so `<=> false` is FALSE, and standard SQL returns only 1. Here `exists <=> false` is TRUE for col1 = 9, the level-2 IN holds for c1 = 9, and every row of t1 comes back. With the decorrelation conf disabled the residual is planned three-valued and the answer is 1, so the two configurations disagree on this shape.
This extends an [existing discussion](https://github.com/apache/spark/pull/58656#issuecomment-6332638268).
See **Shared repair plan 1** in the review body.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -184,8 +188,20 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Use EXISTS if performance matters to you.
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ // The hoisted correlated predicates may carry existential sub-queries that only the
+ // sub-query plan can evaluate, see rewriteExistentialExprInJoinCondition.
+ // This arm returns a bare Join, which this rule does not offer to its own handling of
Review Comment:
**Nit (P3):** This only holds while NOT IN is the last sub-query conjunct. The Filter arm folds every conjunct with `withSubquery.foldLeft`, and a later EXISTS or IN arm wraps this bare anti join as `p` in `Project(p.output, Join(p, ...))`, so the top-down pass reaches it in the `case j: Join` arm, which under the default configuration rewrites the residual into a two-valued `exists`. With t1(a) = 1, 2, 3; t2(c1) = 1, 8, 9; t3n(col1) = 3, NULL:
```sql
SELECT * FROM t1
WHERE a NOT IN (SELECT c1 FROM t2 WHERE c1 > a OR (c1 IN (SELECT col1 FROM t3n)) <=> false)
AND EXISTS (SELECT 1 FROM t2 WHERE c1 >= a)
```
drops row 1, where standard SQL returns 1, 2, 3. The merge base built the same plan, so the behavior is not new; the comment's claim that the residual stays three-valued is.
**Verification:**
- **Inspection:** The comment's stated safety condition matches the fold: a NOT IN followed by another sub-query conjunct is described as reachable by the join-condition rewrite.
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,625 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. With EXISTS the rewrite returns a Project over the join, which that handling
+ // reaches under the default configuration and rejects with
+ // UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION instead, so the query would still
+ // fail but on a different condition than the one asserted here.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: existential subquery carried by a rewritten nested subquery") {
Review Comment:
**Nit (P3):** This is the only test that reaches `rewriteCarriedExistentialExprs`, and it puts the outer EXISTS as a top-level conjunct. The router now also runs on the existence joins built by the Filter-predicate arm, `handleUnaryNode` and the join-condition arm. For example, with `decorrelatePredicateSubqueriesInJoinPredicate.enabled=false`, `SELECT * FROM t1 WHERE a > 100 OR EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t4 WHERE k = c1))` now keeps `k = c1` because the carried IN is routed onto t2, and under the default configuration a carried sub-query that references both inputs is now rejected by the router with a different condition. If the routing stopped applying on those routes, every test would stay green. Cases with the outer predicate under OR, in a SELECT list, and in a user-written join condition, under both values of the conf, would pin it.
**Verification:**
- **Regression:** If the carried routing stopped applying on these routes, the non-default configuration returns extra rows and the new cases fail.
- **Behavior:** With correlation that decides the answer, each route returns the standard SQL rows under both configuration values.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala:
##########
@@ -161,4 +163,124 @@ class RewriteSubquerySuite extends PlanTest {
val existsExpected = LocalRelation(existsQuery.analyze.output).analyze
comparePlans(existsOptimized, existsExpected)
}
+
+ test("SPARK-59351: nested subquery of an EXISTS whose plan shares attributes with the outer " +
+ "plan") {
+ // The EXISTS arms route the nested subquery against the raw subquery plan, while the IN arms
+ // route it against the plan dedupSubqueryOnSelfJoin has already separated from the outer
+ // plan. A self-join can leave an ExprId shared between the two plans for the optimizer
+ // (SPARK-21835), and the analyzer is not involved here, so build that state directly: `a` is
+ // the same attribute in the outer and in the subquery plan.
+ val a = $"a".int
+ val outer = LocalRelation(a, $"b".int)
+ val sub = LocalRelation(a)
+ val nested = LocalRelation($"col1".int)
+ assert(outer.outputSet.intersect(sub.outputSet).nonEmpty,
+ "the two plans must share an attribute for this test to mean anything")
+
+ // The nested subquery references `a`, which only the subquery plan is meant to give it, and
+ // is correlated to it, so it must be rewritten against the subquery plan. Classifying `a` as
Review Comment:
**Nit (P3):** This still describes the 'counts for the outer plan as well' alternative, while the paragraph below and the `planReferences` scaladoc describe the swap, under which `subPlanOnly` and `bothPlans` are both false and the nested sub-query is declined and left in the semi join condition. Rewording this sentence in terms of the swap, for example 'Attributing `a` to the outer plan instead would decline the nested subquery and leave it in the semi join condition', would make the fixture and the scaladoc describe the same alternative.
**Verification:**
- **Inspection:** The fixture comment's alternative and outcome match the scaladoc's swap reading and the exclusive probes in planReferences.
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,625 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. With EXISTS the rewrite returns a Project over the join, which that handling
+ // reaches under the default configuration and rejects with
+ // UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION instead, so the query would still
+ // fail but on a different condition than the one asserted here.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: existential subquery carried by a rewritten nested subquery") {
+ // The nested IN over t4 is owned by the subquery plan, so it is rewritten into an existence
+ // join and the condition PullupCorrelatedPredicates hoisted into it is grafted onto that
+ // join. That condition carries a third-level IN correlated to t4, which references neither
+ // the outer nor the subquery plan and so is routed by neither of the guards above. It has to
+ // be rewritten against the inputs of the join just built: left in place, PlanSubqueries plans
+ // it as an in-subquery filter and drops its own correlated predicate col1 = k, which returns
+ // every row of t1 instead of the one row where the correlation holds.
+ withTempView("t1", "t2", "t4", "t5") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ // No k equals any c1, so only the third-level IN can satisfy the hoisted disjunction, and
+ // its correlated predicate col1 = k is false for every row of t4: dropping it makes the
+ // level-2 IN hold for c1 = 1 and returns 1, 2 and 3.
+ Seq((1, 100), (9, 7)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(1).toDF("col1").persist().createOrReplaceTempView("t5")
+ val query =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t4
+ | WHERE k = c1 OR col1 IN (SELECT col1 FROM t5 WHERE col1 = k)))""".stripMargin
+ // The routing must not depend on the rewrite of predicate subqueries in join conditions,
+ // which is what used to make these two configurations disagree: it reached the grafted
+ // condition only when enabled.
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswerAndPlan(query, Row(1) :: Nil)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: NOT IN keeps an uncorrelated three-valued residual") {
+ // The NOT IN rewrite returns a bare Join, which this rule does not offer to its own handling
+ // of predicate subqueries in join conditions, so an uncorrelated nested IN left in the
+ // anti-join condition stays there and InSubqueryExec evaluates it three-valued. No existence
+ // join is built for it and it has no hoisted condition to lose, so it must not be rejected:
+ // these queries returned the standard SQL answer before SPARK-59351 and must keep doing so.
+ withTempView("t1", "t2", "t3n", "t4n") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ // The NULL matches no c1, so c1 IN (SELECT col1 FROM t3n) is unknown for every row of t2.
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // Nullable col1 again, but correlated through k = c1.
+ Seq((Some(3), 1), (None, 8)).toDF("col1", "k").persist().createOrReplaceTempView("t4n")
+
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> false",
+ "(c1 IN (SELECT col1 FROM t3n)) IS NULL",
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), true)",
+ "(c1 NOT IN (SELECT col1 FROM t3n)) <=> false").foreach { predicate =>
+ // Every such predicate is FALSE for every row of t2, so the subquery reduces to a = c1
+ // and only 2 and 3 survive the NOT IN. Rejecting instead, or rewriting the nested IN into
+ // a two-valued exists, would change this.
+ checkAnswer(
+ sql(s"SELECT * FROM t1 WHERE a NOT IN (SELECT c1 FROM t2 WHERE a = c1 OR $predicate)"),
Review Comment:
**Nit (P3):** With the hoisted disjunct `a = c1` repeating the NOT IN key, this query cannot depend on `$predicate`: S(1) contains 1 whatever the predicate is, and for a = 2 or 3, S(a) holds only non-null values other than a. So the answer is 2, 3 whether the residual is evaluated three-valued, rewritten into a two-valued `exists`, or dropped; only a rejection is detected. The comment above also says each predicate is FALSE for every row of t2, but `IS NULL` and `COALESCE(..., true)` are TRUE there. The truth-only case at the end asserts no plan, so 'rewritten, not kept' is unchecked too. A hoisted predicate that is not the key equality, such as `c1 > a OR $predicate` as in query5 of the first new test, over data where three-valued and two-valued evaluation differ, would make each case discriminate.
This extends existing discussions [1](https://github.com/apache/spark/pull/58656#issuecomment-5731841503), [2](https://github.com/apache/spark/pull/58656#pullrequestreview-5247095515).
**Verification:**
- **Regression:** With the residual replaced by a two-valued rewrite, at least one asserted answer in each residual case changes.
- **Behavior:** The rebuilt cases return the standard SQL answers, including for NULLs in the nested list.
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,625 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. With EXISTS the rewrite returns a Project over the join, which that handling
Review Comment:
**Nit (P3):** As written, this describes current behavior, but with EXISTS as the outer predicate the EXISTS arm reaches the same rejection in `rewriteExistentialExprInSubqueryPlan`, which raises `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY` before any join exists. The join-condition handling would reject with `UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION` only if that rejection were removed. Saying so explicitly, for example 'were this rejection removed, the EXISTS form would ...', or dropping the EXISTS sentence, would make the rationale accurate.
**Verification:**
- **Inspection:** The rationale matches current behavior, where the EXISTS form of the query fails with NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY, and states any other outcome only as a counterfactual.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,476 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. The two plans share no ExprId in any
+ * shape reached by the suite: `DeduplicateRelations` has given the sub-query its own, and the
+ * EXISTS arms classify against the raw sub-query plan while the IN arms classify against the
+ * plan `dedupSubqueryOnSelfJoin` has already separated from the outer plan. The order is
+ * therefore defensive rather than something a query exercises, and it is the safe one: a
+ * nested sub-query resolves in the scope of the sub-query it is nested in, so an attribute
+ * both plans produce is one of that sub-query's own, and attributing it to `outerPlan` instead
+ * would let a single shared attribute satisfy both sides of the classification and have a
+ * nested sub-query referencing only the inner query rejected as referencing both.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite
+ * without rejecting the sub-query, as a sub-query referencing neither plan can be left in the
+ * join condition unharmed.
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
Review Comment:
Thanks, the correlated carried descendant is routed now and both configurations agree on that query. The second case still fails on 4168a32841b: with t4(col1, k) = (9, 7) and t3n(x) = (3, NULL), rewriteCarriedExistentialExprs leaves the uncorrelated `(col1 IN (SELECT x FROM t3n)) <=> false` in place, and under the default configuration the join-condition arm rewrites it into a two-valued exists, so every row of t1 is returned instead of 1; see the comment on rewriteCarriedExistentialExprs in optimizer/subquery.scala.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4257086397","thread_id":"inline:4257086397","verdict_sha256":"3bb54f2a3a4b20553e9d3ca5caee9c4a8fc61921fc05f7c2363e10d0f4e5e7b0"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,476 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. The two plans share no ExprId in any
Review Comment:
Thanks, the planReferences scaladoc now uses the swap reading in both paragraphs. The fixture comment in RewriteSubquerySuite still says classifying `a` as an outer reference as well would report it as referencing both plans; see the comment there.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4257086406","thread_id":"inline:4257086406","verdict_sha256":"3bb54f2a3a4b20553e9d3ca5caee9c4a8fc61921fc05f7c2363e10d0f4e5e7b0"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -184,8 +188,12 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Use EXISTS if performance matters to you.
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ // The hoisted correlated predicates may carry existential sub-queries that only the
+ // sub-query plan can evaluate, see rewriteExistentialExprInJoinCondition.
+ val (subConditions, newSub) =
+ rewriteExistentialExprInSubqueryPlan(conditions, p, dedupSub)
Review Comment:
Confirmed: the NOT IN arm keeps the uncorrelated residual and the four shapes return 2, 3 as at the merge base, while the correlated ones are still rejected. Two follow-ups on the new comment and test are in this review: the bare-Join premise does not hold when the NOT IN is not the last sub-query conjunct, and the hoisted `a = c1` means the test's rows cannot tell a three-valued residual from a two-valued rewrite.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4281437494","thread_id":"inline:4281437494","verdict_sha256":"3bb54f2a3a4b20553e9d3ca5caee9c4a8fc61921fc05f7c2363e10d0f4e5e7b0"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4356951704
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,568 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is deliberately left where it is. It has no hoisted condition to
+ * lose, and `InSubqueryExec` evaluates it with the three-valued semantics that the `exists`
+ * attribute of an existence join cannot carry.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ // Only a correlated sub-query has to be consumed here, see above.
+ val carried = condition.collect {
+ case sq @ (_: Exists | _: InSubquery) if hasCorrelatedCondition(sq) => sq
+ }
+ if (carried.isEmpty) {
+ return join
+ }
+ // Where the value of a carried IN is observed beyond being TRUE, an existence join cannot
+ // stand in for it, exactly as on the routes above.
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+
+ val rejected = carried.filter { sq =>
+ observesUnknown(sq) ||
+ (referencesInput(sq, join.left) && referencesInput(sq, join.right))
+ }
+ if (rejected.nonEmpty) {
+ // A sub-query that needs both inputs can be rewritten against neither, and one whose
+ // unknown is observable cannot become an `exists` attribute at all.
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(rejected)
+ }
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ carried.filter(sq => isInput(sq) && !rejected.contains(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) = rewriteExistentialExprWithAttrs(Seq(sq), p)
Review Comment:
Confirmed and fixed, and your diagnosis of the cause is exact: `_: InSubquery` matched the IN *inside* the `Not`, so `route` handed the bare IN to the positive-IN arm and the null-aware condition was never built.
Verified against your fixture:
| | result |
|---|---|
| before, both configurations | `1, 2` |
| **now, both configurations** | **no rows** |
| DuckDB | no rows |
| merge base | `INTERNAL_ERROR` |
The collection is now done by `collectCarriedExistentialExprs`, a small recursive walk rather than a `collect`, which takes a NOT IN as the `Not` over the `InSubquery` and does not descend into a collected node. A plain `collect` would have matched both the `Not` and the inner IN and classified the same sub-query twice.
That also fixes the second consequence you noted: `unknownSensitiveInSubqueries` reports the `Not`, and `observesUnknown` now compares against the same node, so a carried `(col1 NOT IN (...)) <=> false` is rejected rather than routed. `hasCorrelatedCondition` gained a `Not(in: InSubquery)` case so the unit is classified by the IN it negates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6380316971
Thanks for re-reviewing it @cloud-fan, I have addressed the previously open discussion and the new ones as well, PTAL again and do let know if further changes are required, TIA!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114403057
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,206 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) && !effectivelyReferencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ case Not(child) => unknownSensitiveInSubqueries(child, inPredicate)
+ case in: InSubquery =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ // The join condition of a sub-query expression is a predicate, evaluated by the join that
+ // the sub-query is rewritten into.
+ case sq: SubqueryExpression =>
+ sq.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ case other =>
+ other.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = false))
+ }
+
+ /**
+ * Returns true if `e` is a positive IN sub-query whose row comparison can evaluate to unknown,
+ * that is one that can return NULL rather than only TRUE or FALSE. An existence join cannot
+ * represent that third value, see [[rewriteExistentialExprInJoinCondition]]. NOT IN is excluded,
+ * as [[rewriteExistentialExprWithAttrs]] gives it a null-aware join condition of its own.
+ */
+ private def inSubqueryMayBeUnknown(e: Expression): Boolean = e match {
+ case InSubquery(values, listQuery) =>
+ values.zip(listQuery.plan.output).exists { case (v, o) => v.nullable || o.nullable }
Review Comment:
Added: a nullable compared value against a non-nullable, non-matching list query. Deleting
`v.nullable ||` fails that test.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,206 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) && !effectivelyReferencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ case Not(child) => unknownSensitiveInSubqueries(child, inPredicate)
+ case in: InSubquery =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ // The join condition of a sub-query expression is a predicate, evaluated by the join that
+ // the sub-query is rewritten into.
+ case sq: SubqueryExpression =>
+ sq.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ case other =>
+ other.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = false))
+ }
+
+ /**
+ * Returns true if `e` is a positive IN sub-query whose row comparison can evaluate to unknown,
+ * that is one that can return NULL rather than only TRUE or FALSE. An existence join cannot
+ * represent that third value, see [[rewriteExistentialExprInJoinCondition]]. NOT IN is excluded,
+ * as [[rewriteExistentialExprWithAttrs]] gives it a null-aware join condition of its own.
+ */
+ private def inSubqueryMayBeUnknown(e: Expression): Boolean = e match {
+ case InSubquery(values, listQuery) =>
+ values.zip(listQuery.plan.output).exists { case (v, o) => v.nullable || o.nullable }
Review Comment:
Added a nullable compared value against a non-nullable, non-matching list query. Deleting
`v.nullable ||` fails that test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114427431
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +158,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
Review Comment:
Confirmed. I reproduced it in a Catalyst test rather than through SQL, since
`DeduplicateRelations` removes the ExprId overlap from every analyzed query I could write: with the
outer and the subquery plan sharing `a`, a nested subquery that references only the inner query is
reported as `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY`.
I took the second half of your instruction — "or otherwise ensure routing uses the deduplicated
subplan" — rather than moving the deduplication, for the reason in your own Risks line. For the
EXISTS arms the two are not equivalent: `dedupSubqueryOnSelfJoin` aliases the subquery plan's `a` to
a fresh ExprId while the hoisted `conditions` still say `a`, and since the semi join's left side is
the outer plan, which still produces `a`, the condition would rebind to the outer attribute. That
turns a correlated predicate into a self-reference silently, with no error, which is worse than the
misclassification it would fix. The IN arms escape it only because they rebuild `inConditions` from
`dedupSub.output`; the hoisted `conditions` they pass along are equally exposed.
So the classification now behaves as though the subplan had been deduplicated:
`effectivelyReferencesPlanOnly` attributes an attribute that both plans produce to the subquery
plan, because a nested subquery resolves in the scope of the subquery it sits in, and only an
attribute the outer plan alone produces counts as an outer reference. That is the classification the
IN arms get from their deduplicated plan, with no change to deduplication or to the errors it
raises.
Against your success criteria:
* *"A genuinely both-side correlated nested subquery is still rejected"* — holds, and the
`SubquerySuite` case still fails if the rejection is removed.
* *"Routed to that plan despite pre-dedup ExprId overlap"* — holds for the routing: the existence
join is built on the subquery plan and the nested subquery is no longer misclassified. The query
still does not succeed, though. `buildJoin` hands the semi join condition to
`dedupSubqueryOnSelfJoin`, which has always rejected a condition naming a duplicated attribute, so
it ends at `_LEGACY_ERROR_TEMP_1212` — the pre-existing behaviour dongjoon describes in the
dedup-anchor comment. The user-visible change is a better error, not a working query.
Making that shape succeed is the single preparation step you describe, and it needs the hoisted
condition rewritten along with the plan, which changes a pre-existing error path for all four arms.
I have kept it out so this PR stays a fix for the attribute-routing bug. Happy to do it here if you
want it in scope; otherwise I will file it with the repro from the new test.
The new test is in `RewriteSubquerySuite`, covering a self-correlated EXISTS with a nested subquery
under ExprId overlap; it fails with the 0A000 misclassification if the classification change is
reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114359422
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) && referencesPlan(sq, subPlan) &&
+ referencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => referencesPlan(sq, subPlan) && !referencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => referencesPlan(e, subPlan) && !referencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ case Not(child) => unknownSensitiveInSubqueries(child, inPredicate)
+ case in: InSubquery =>
Review Comment:
Confirmed, and it was the silent wrong answer you describe. `InSubquery` is a `Predicate`, so it
never reached the `SubqueryExpression` arm that descends, and the walk stopped at the level-2 IN.
`unknownSensitiveInSubqueries` now descends through an IN sub-query it does not collect, into the
values it compares and into its list query, in a helper shared with the `Not(InSubquery)` arm. Your
query is no longer answered from a two-valued bit; the level-2 IN is seen and, being in a value
position, rejected. Added as a regression test, which fails without the descent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5891452714
> I would keep full self-join preparation and the pre-existing dedupSubqueryOnSelfJoin asymmetry as a follow-up: current supported SQL inputs are analyzer-deduplicated, and the remaining conflict failure predates this PR. The effective-reference invariant is appropriate because it is derived from the compared values and hoisted conditions the rewrite actually consumes; I do not see a supported reused-ExprId counterexample on this head. The two error names are reasonable. Please file the released outer/projection three-valued-IN behavior as a separate JIRA. Also update the PR description's both-plan example to name NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY.
Thanks for the comments @cloud-fan, I have addressed all of them, and have update PR description as well - PTAL again. Also have opened a new JIRA [SPARK-59862](https://issues.apache.org/jira/browse/SPARK-59862) to track `three-valued-IN behavior`, I will start working on this shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4139416462
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) => collectOrDescend(not, in, truthOnly, reported)
+ case Not(child) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = false, reported)
+ case in: InSubquery => collectOrDescend(in, in, truthOnly, reported)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = true, reported))
+ sq.getOuterAttrs.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ case other =>
+ other.children.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ }
+
+ /**
+ * Appends `report` to `reported` if the IN sub-query `in` would lose an observable unknown, and
+ * otherwise descends into the values it compares and into its list query. The descent matters
+ * because [[InSubquery]] is a [[Predicate]] rather than a [[SubqueryExpression]], so an IN
+ * sub-query nested in the values or in the hoisted join condition of another one is only reached
+ * from here.
+ */
+ private def collectOrDescend(
+ report: Expression,
+ in: InSubquery,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = {
+ if (!truthOnly && inSubqueryMayBeUnknown(in)) {
+ reported += report
+ } else {
+ in.values.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ collectUnknownSensitiveInSubqueries(in.query, truthOnly, reported)
+ }
+ }
+
+ /**
+ * Returns true if the row comparison of the IN sub-query `in` can evaluate to unknown, that is
+ * if it can return NULL rather than only TRUE or FALSE. An existence join cannot represent that
+ * third value, see [[rewriteExistentialExprInJoinCondition]].
+ *
+ * This deliberately does not use `InSubquery.nullable`, which under
+ * `spark.sql.legacy.inSubqueryNullability` (SPARK-43413) reports only the nullability of the
+ * compared values and ignores that of the list query's columns. What is protected here is the
+ * three-valued result the rewrite would drop, which does not depend on that configuration, so
+ * the nullability is re-derived from both sides.
+ */
+ private def inSubqueryMayBeUnknown(in: InSubquery): Boolean = {
+ in.values.zip(in.query.childOutputs).exists { case (v, o) => v.nullable || o.nullable }
Review Comment:
**Nit (P3):** This `zip` eagerly constructs every value/output pair before `exists` can short-circuit. For a wide row-valued IN whose first pair is nullable, the optimizer still allocates and traverses a collection proportional to the full arity.
**Verification:**
- **Inspection:** Confirm that the nullable-pair predicate consumes pairs lazily and has no strict intermediate pair collection.
- **Behavior:** Cover scalar and multi-column IN cases with nullable values and nullable list outputs and verify unchanged UNKNOWN classification.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +159,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftSemi, joinCond, subHint)
Project(p.output, join)
case (p, Not(Exists(sub, _, _, conditions, subHint))) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftAnti, joinCond, subHint)
Project(p.output, join)
case (p, InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val (joinCond, outerPlan) = rewriteExistentialExpr(inConditions ++ conditions, p)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(inConditions ++ conditions, p, dedupSub)
Review Comment:
**Blocking (P1):** Routing the nested subquery here can make `ExistenceJoin(B, C)` the build side of the enclosing semi join, but its generic product estimate can be much larger than B even though the operator preserves exactly B's rows. That can incorrectly disqualify broadcast and force a shuffle and sort of the large outer input.
**Recommended change:** Teach both statistics modes that ExistenceJoin preserves the left row count, adds only its Boolean exists attribute to the left output width, and does not multiply by right cardinality; add statistics and nested-subquery plan-selection regressions.
**Why this works:** Add an explicit ExistenceJoin estimate to the CBO join estimator and an explicit left-derived estimate to the size-only join visitor. Compute size from the left cardinality or size and the actual ExistenceJoin output width, propagate only valid left-side attribute statistics, and verify that an ExistenceJoin-wrapped build side remains broadcast-eligible when its left input is below the threshold regardless of the right estimate.
**Scope:** Correct ExistenceJoin statistics consistently in both logical statistics modes and pin the downstream broadcast decision.
**Compatibility:** ExistenceJoin still returns every left row with one Boolean match indicator, and all predicate-subquery routing and result semantics remain unchanged.
**Risks:** Copying the left size verbatim would omit the added Boolean output width. Propagating right-side column statistics would describe attributes that ExistenceJoin does not output. Changing generic join fallback behavior could alter estimates for unrelated join types.
**Constraints:** ExistenceJoin emits exactly one output row per left input row and appends one non-nullable Boolean attribute. The estimate must remain valid when right-side row counts are absent or zero. CBO-enabled and CBO-disabled plans must agree on the left-preserving cardinality invariant.
**Success:** An ExistenceJoin estimate never scales its row count or size by the right child's cardinality. With CBO enabled, the estimated row count equals the left row count and output size includes the appended Boolean attribute. With CBO disabled, sizeInBytes is derived from the left input and ExistenceJoin output width rather than the product of both children. A nested-subquery ExistenceJoin on an otherwise broadcastable build side does not disqualify the enclosing join solely because of the nested right estimate.
**Verification:**
- **Behavior:** Verify left-preserving ExistenceJoin row count, output size, and attribute-stat propagation for nonempty and empty right inputs with CBO enabled.
- **Compatibility:** Verify the same left-preserving size invariant with CBO disabled and confirm unrelated join estimates remain unchanged.
- **Regression:** Construct a nested inner-routed existence join whose left child is below the broadcast threshold and whose right estimate would make the old product exceed it, then verify the enclosing equi semi join retains the broadcast plan.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
Review Comment:
**Nit (P3):** This is not the only configuration under which the subquery survives. With decorrelation enabled but uncorrelated-IN optimization in join conditions disabled, the earlier relevant-subquery filter also leaves this expression in the join condition.
**Verification:**
- **Inspection:** Trace the documented alternatives through the relevant-subquery filter and downstream join-condition rewrite and confirm neither supported configuration branch is omitted.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) => collectOrDescend(not, in, truthOnly, reported)
+ case Not(child) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = false, reported)
+ case in: InSubquery => collectOrDescend(in, in, truthOnly, reported)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = true, reported))
+ sq.getOuterAttrs.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ case other =>
+ other.children.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ }
+
+ /**
+ * Appends `report` to `reported` if the IN sub-query `in` would lose an observable unknown, and
+ * otherwise descends into the values it compares and into its list query. The descent matters
+ * because [[InSubquery]] is a [[Predicate]] rather than a [[SubqueryExpression]], so an IN
+ * sub-query nested in the values or in the hoisted join condition of another one is only reached
+ * from here.
+ */
+ private def collectOrDescend(
+ report: Expression,
+ in: InSubquery,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = {
+ if (!truthOnly && inSubqueryMayBeUnknown(in)) {
+ reported += report
+ } else {
+ in.values.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ collectUnknownSensitiveInSubqueries(in.query, truthOnly, reported)
+ }
+ }
+
+ /**
+ * Returns true if the row comparison of the IN sub-query `in` can evaluate to unknown, that is
+ * if it can return NULL rather than only TRUE or FALSE. An existence join cannot represent that
+ * third value, see [[rewriteExistentialExprInJoinCondition]].
+ *
+ * This deliberately does not use `InSubquery.nullable`, which under
+ * `spark.sql.legacy.inSubqueryNullability` (SPARK-43413) reports only the nullability of the
+ * compared values and ignores that of the list query's columns. What is protected here is the
+ * three-valued result the rewrite would drop, which does not depend on that configuration, so
+ * the nullability is re-derived from both sides.
+ */
+ private def inSubqueryMayBeUnknown(in: InSubquery): Boolean = {
+ in.values.zip(in.query.childOutputs).exists { case (v, o) => v.nullable || o.nullable }
+ }
+
+ /**
+ * Returns true if `e` is an existential sub-query expression that still carries a correlated
+ * condition, that is one that [[PullupCorrelatedPredicates]] hoisted into it and that would be
+ * lost were the sub-query left in a join condition.
+ *
+ * This is decided from that condition rather than from `isCorrelated`, which reports the outer
Review Comment:
**Nit (P3):** `isCorrelated` does not report the outer attributes; it is the Boolean `outerAttrs.nonEmpty`. Describing it as the attribute collection obscures why this helper checks the surviving join condition instead.
**Verification:**
- **Inspection:** Compare the revised explanation with the SubqueryExpression correlation accessors and confirm each identifier's value and role are stated accurately.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
Review Comment:
**Nit (P3):** This avoids materializing the `intersect` result, but it is not allocation-free: `AttributeSet` iteration and membership probes construct wrappers. The stronger claim can mislead later performance work on this hot classification helper.
**Verification:**
- **Inspection:** Check the revised statement against AttributeSet iterator, contains, and intersect behavior and confirm every allocation claim is supportable.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
Review Comment:
**Nit (P3):** This manual replacement now owns the transform framework's origin and tag propagation, but the tests do not exercise either guarantee. Dropping `withOrigin` or `copyTagsFrom` would leave the current result and plan checks green while moving diagnostics to the enclosing operator or losing expression metadata.
**Verification:**
- **Regression:** Cause a runtime error from a successfully rewritten nested condition and assert that the reported SQL fragment and coordinates belong to that nested condition rather than the enclosing operator.
- **Regression:** Attach a TreeNode tag to a rewriteable nested expression and verify the produced replacement retains the tag.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) => collectOrDescend(not, in, truthOnly, reported)
+ case Not(child) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = false, reported)
+ case in: InSubquery => collectOrDescend(in, in, truthOnly, reported)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = true, reported))
+ sq.getOuterAttrs.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ case other =>
+ other.children.foreach(
Review Comment:
**Non-blocking (P2):** `COALESCE(nullableIn, false)` maps both UNKNOWN and FALSE to false, so this is still a truth-only context and the existence-bit rewrite is exact. Falling through to the generic child traversal marks it value-sensitive and raises the new unsupported error for a valid query.
**Verification:**
- **Regression:** Exercise a hoisted nullable IN under a false-default COALESCE and verify both the expected rows and placement of the rewritten existence join.
- **Behavior:** Exercise comparable true-default and null-observing wrappers and verify they still raise the UNKNOWN-result rejection.
- **Compatibility:** Verify the result under both legacy IN-subquery nullability settings and preserve direct NOT IN handling.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
Review Comment:
**Nit (P3):** The adjacent ownership checks each recompute `effectiveReferences(sq)`, rebuilding and traversing the same reference set. The UNKNOWN filter and rewrite predicate repeat the pattern, so classification duplicates work proportional to the nested expression's referenced attributes.
**Verification:**
- **Inspection:** Trace every classification phase for one nested expression and confirm effective references are derived once and reused rather than reconstructed by each Boolean probe.
- **Compatibility:** Verify effective ownership results for inner-only, outer-only, both-plan, unreferenced, shared-ExprId, and simplified-away-correlation shapes remain unchanged.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
Review Comment:
**Nit (P3):** The wording reverses the premise: a subquery that references both plans cannot be rewritten on either side, which is why it must remain in the join condition. The same contradictory phrasing also appears in the added test comment.
**Verification:**
- **Inspection:** Read the production and test explanations with their adjacent ownership predicates and verify that neither sentence reverses the capability claim.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) && referencesPlan(sq, subPlan) &&
Review Comment:
This now uses the surviving values and hoisted conditions for ownership and hasCorrelatedCondition for rejection; the false-AND regression confirms retained outerAttrs alone do not reject the query.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4057052746","thread_id":"inline:4057052746","verdict_sha256":"15b8678aea32e2f6c0ae8bee1554ad3f1ae7e6de47363ee749ece337d2c0f9ac"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
Review Comment:
The Scaladoc now names spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled exactly.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4057052748","thread_id":"inline:4057052748","verdict_sha256":"15b8678aea32e2f6c0ae8bee1554ad3f1ae7e6de47363ee749ece337d2c0f9ac"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +158,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
Review Comment:
Routing now treats an ExprId shared by both plans as subquery-scoped, matching the post-dedup IN classification without rebinding the hoisted condition; the Catalyst regression fails on the former false both-side rejection.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4068594134","thread_id":"inline:4068594134","verdict_sha256":"15b8678aea32e2f6c0ae8bee1554ad3f1ae7e6de47363ee749ece337d2c0f9ac"} -->
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3227,284 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // Checks the result, and that every node of the optimized plan can produce the attributes
+ // it references. The latter is what this fix is about: an existence join whose condition
+ // references an attribute produced by neither of its children can still return the right
+ // answer when the nested relation is empty at runtime, because the invalid condition is
+ // then never bound. checkAnswer alone would not catch it, as the missing input checks it
+ // runs only look at the root of the plan.
+ def checkAnswerAndPlan(query: String, expected: Seq[Row]): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ checkAnswer(df, expected)
+ }
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // would drop its correlated predicate col1 = c1 and return an extra row, so it is rejected.
Review Comment:
The stale result-change statement is removed; the NOT IN case now claims only the dropped correlated predicate and points to the EXISTS form for the extra-row demonstration.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4068594144","thread_id":"inline:4068594144","verdict_sha256":"15b8678aea32e2f6c0ae8bee1554ad3f1ae7e6de47363ee749ece337d2c0f9ac"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,14 +405,26 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
+ def rewrite(expr: Expression): Expression = {
Review Comment:
Each replacement now runs under sq.origin and calls copyTagsFrom(sq), while retaining the custom do-not-descend behavior.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4068594150","thread_id":"inline:4068594150","verdict_sha256":"15b8678aea32e2f6c0ae8bee1554ad3f1ae7e6de47363ee749ece337d2c0f9ac"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4286964125
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,481 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. A nested sub-query resolves in the scope
+ * of the sub-query it is nested in, so an attribute both plans produce is one of that
+ * sub-query's own; attributing it to `outerPlan` instead would let a single shared attribute
Review Comment:
Agreed, and I have picked the swap reading, since that is the mutation the probe order actually admits.
Under a swap the probes stay exclusive, so a shared attribute becomes outer-only: `subPlanOnly` and `bothPlans` are both false and the nested subquery is **declined and left in the join condition** rather than routed to the subquery plan. It is not "rejected as referencing both" -- that claim needed the "counts for the outer plan as well" reading, which the code does not implement. Both paragraphs and the fixture comment now say the same thing, and the fixture-outcome sentence is kept, since it is true under the swap reading: either order ends in the same error, because `dedupSubqueryOnSelfJoin`'s conflict check reports the duplicated attribute the hoisted condition mentions.
The "neither plan" paragraph also needed updating for a different reason: it said such a subquery "can be left in the join condition unharmed", which is the assumption `r4257086397` disproved. It now points at `rewriteCarriedExistentialExprs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4286938924
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/JoinEstimationSuite.scala:
##########
@@ -466,6 +466,51 @@ class JoinEstimationSuite extends StatsEstimationTestBase {
}
}
+ test("existence join") {
+ // table2 (key-1-2 int, key-2-4 int): (1, 2), (2, 3), (2, 4)
+ // table3 (key-1-2 int, key-2-3 int): (1, 2), (2, 3)
+ // An existence join returns every row of its left child with one extra Boolean attribute, so
+ // its cardinality is the left child's and its width is the left child's plus that attribute.
+ // The right child's cardinality must not enter either, as the generic product estimate would
+ // have it, since that can disqualify a broadcast of an otherwise small build side.
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val join = Join(table2, table3, ExistenceJoin(exists),
+ Some(EqualTo(nameToAttr("key-2-4"), nameToAttr("key-2-3"))), JoinHint.NONE)
+
+ val expectedStats = Statistics(
+ // Three rows of the left child, each widened by the Boolean, whose default size is 1.
+ sizeInBytes = 3 * (8 + 4 * 2 + 1),
+ rowCount = Some(3),
+ // Only the left child's column stats: the right child's describe attributes that an
+ // existence join does not output.
+ attributeStats = AttributeMap(Seq(nameToColInfo("key-1-2"), nameToColInfo("key-2-4"))))
+ assert(join.stats == expectedStats)
+ }
+
+ test("existence join with an empty or unestimated right child") {
+ // Only the left child's row count enters the estimate, so a right child with no row count,
Review Comment:
You are right on both counts, and this was my error. `StatsTestPlan` takes a required `rowCount: BigInt` and its `computeStats` always returns `rowCount = Some(rowCount)`, so neither fixture had no row count -- `emptyRight` had 0 and `noStatsRight` had 2, and only their column statistics were missing. The test name and comment claimed a case the test did not build.
The test is renamed `existence join with a right child that has no row count or no statistics` and now builds a `NoRowCountPlan` whose `computeStats` omits `rowCount` entirely, alongside the zero-row-count, no-column-statistics case.
Checked against the mutation you named: tightening `rowCountsExist(join.left)` to `rowCountsExist(join.left, join.right)` now fails this test, and only this test, out of the 44 in `JoinEstimationSuite` and `BasicStatsEstimationSuite`. Before the change it stayed green, as you said it would.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4281440369
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
Review Comment:
Confirmed: both the scaladoc and the test comment now read that the subquery cannot be rewritten on either side and therefore stays in the join condition.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4139416489","thread_id":"inline:4139416489","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
Review Comment:
Confirmed: the comment now says the helper avoids materializing the intersection and that the membership probes still construct wrappers.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4139416492","thread_id":"inline:4139416492","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
Review Comment:
Confirmed: planReferences classifies each candidate in one pass, and the IdentityHashMap memo shares the result across the both-plans rejection, the unknown filter and canRewrite.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4139416496","thread_id":"inline:4139416496","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
Review Comment:
Confirmed: RewriteSubquerySuite now pins the origin and tags on the Exists replacement and the origins of the synthesized IN and NOT IN conditions, and it checks that the reused hoisted condition keeps its own origin.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4139416499","thread_id":"inline:4139416499","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) => collectOrDescend(not, in, truthOnly, reported)
+ case Not(child) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = false, reported)
+ case in: InSubquery => collectOrDescend(in, in, truthOnly, reported)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = true, reported))
+ sq.getOuterAttrs.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ case other =>
+ other.children.foreach(
Review Comment:
Confirmed: COALESCE with a FALSE default now passes the truth-only context to its first argument. The suite covers the false default as rewritten and the true default as rejected.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4139416502","thread_id":"inline:4139416502","verdict_sha256":"666f73c8705c7fc517a3755f43545348abf6bf7b1b48ab8a7138d74dc04fbaa3"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6250929718
@cloud-fan Gentle Ping!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4274524148
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,375 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: nested IN subquery whose result can be unknown") {
+ withTempView("t1", "t2", "t2n", "t3n", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // A nullable compared value, against a subquery column that is not nullable.
+ Seq(Some(1), None).toDF("c1").persist().createOrReplaceTempView("t2n")
+ Seq((1, 1), (9, 9)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ // The NULL matches no value of t2, so c1 IN (SELECT col1 FROM t3n) is unknown there, and
+ // c1 NOT IN (SELECT col1 FROM t3n) is never true.
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // An existence join only records whether a row matched, so its exists attribute is false
+ // where IN is unknown. That is not observable while the value feeds a predicate, but it is
+ // once it reaches anything else, so these are rejected rather than rewritten. Both
+ // configurations are checked, as the rejection must not depend on the rewrite of predicate
+ // subqueries in join conditions.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> false",
+ "(c1 IN (SELECT col1 FROM t3n)) IS NULL",
+ // COALESCE to a non-false default does tell unknown from false.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), true)",
+ "(c1 NOT IN (SELECT col1 FROM t3n)) <=> false").foreach { predicate =>
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)")
+ .collect()
+ }
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT")
+ // A NOT IN must be reported as a NOT IN, not as the IN underneath it.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.startsWith("(NOT ") == predicate.contains("NOT IN"),
+ s"unexpected subqueryExpression for $predicate: $reported")
+ }
+ }
+ }
+
+ // The ordinary case: the value of the nested IN feeds a predicate, where unknown and
+ // false cannot be told apart, so it is still rewritten even though it can be unknown.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t3n))""".stripMargin,
+ Row(1) :: Nil)
+
+ // And a nested IN that cannot be unknown is rewritten even outside a predicate.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3)) <=> false))""".stripMargin,
+ Row(1) :: Row(2) :: Row(3) :: Nil)
+
+ // Only truth is observed by `<=> true`, by an IF predicate and by a CASE WHEN
+ // condition, so those are rewritten rather than rejected. SimplifyConditionals folds
+ // the latter two into the first, so ordinary SQL reaches this position.
+ //
+ // The result branches of a conditional are consumed wherever the conditional is, so
+ // under a filter predicate they are truth-only as well. The last two cases do not
+ // fold, so they exercise that inheritance rather than the rewrites above.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> true",
+ "IF(c1 IN (SELECT col1 FROM t3n), true, false)",
+ "CASE WHEN c1 IN (SELECT col1 FROM t3n) THEN true ELSE false END",
+ "IF(c1 > 0, c1 IN (SELECT col1 FROM t3n), false)",
+ "CASE WHEN c1 > 0 THEN c1 IN (SELECT col1 FROM t3n) ELSE false END",
+ "CASE WHEN c1 > 100 THEN false ELSE c1 IN (SELECT col1 FROM t3n) END",
+ // COALESCE to false maps unknown and false alike, so only truth is observed.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), false)")
+ .foreach { predicate =>
+ checkAnswerAndPlan(
+ s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)",
+ Row(1) :: Nil)
+ }
+
+ // The unknown can come from the compared value rather than from the subquery column.
+ // spark.sql.legacy.inSubqueryNullability makes InSubquery.nullable ignore the
+ // nullability of the list query columns, but it always keeps that of the compared
+ // values, and the guard re-derives both, so this case is rejected under either value.
+ Seq("true", "false").foreach { legacyNullability =>
+ withSQLConf(SQLConf.LEGACY_IN_SUBQUERY_NULLABILITY.key -> legacyNullability) {
+ val nullableValue = intercept[AnalysisException] {
+ sql("""SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2n
+ | WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3)) <=> false))""".stripMargin)
+ .collect()
+ }
+ assert(nullableValue.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT")
+ }
+ }
+
+ // An IN subquery nested in the values or in the hoisted condition of another one is
Review Comment:
Addressed on the current HEAD — apologies for the late reply on this thread.
Confirmed, and the gap was real. Removing
```scala
in.values.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
```
from `collectOrDescend` left the whole suite green, so as far as the tests were concerned the values-side descent was dead code. You are also right about why: `nestedTwoLevels` reaches the inner IN through the outer `ListQuery`'s hoisted `joinCond`, never through `in.values`.
That case is renamed `nestedInJoinCondition` and its comment now claims only the hoisted-condition descent. A second case covers the values position:
```sql
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2
WHERE a = c1
OR (c1 IN (SELECT col1 FROM t3n)) IN (SELECT b FROM tb))
```
with `t1 = (1, 2, 3)`, `t2.c1 = (1, 8, 9)`, `t3n.col1 = (3, NULL)` and `tb.b = (false)`.
`c1 IN (SELECT col1 FROM t3n)` is UNKNOWN for every `c1` of `t2`, and here it is a compared value rather than a predicate, so UNKNOWN and FALSE are distinguishable: `UNKNOWN IN (false)` is UNKNOWN, while the FALSE an existence join would leave in its place gives `false IN (false)` = TRUE. So the IN has to be rejected, and it is.
The case discriminates, which is the point of adding it:
| | with the values descent | without it |
|---|---|---|
| this query | `NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT` | no error, returns `1, 2, 3` |
| `nestedInJoinCondition` | rejected | rejected (still green) |
`1, 2, 3` is the wrong answer you predicted: the OR reduces to `a = c1`, which holds only for `a = 1`. DuckDB returns `[(1,)]` for the same data.
The assertion also pins *which* IN is named, since the enclosing IN is unknown-producing too and only the nested one may be reported:
```scala
assert(reportedNested == "(t2.c1 IN (listquery()))", reportedNested)
```
On the compatibility half of your request: the direct `NOT IN` behaviour and the existing nested join-condition rejection are unchanged — `SubquerySuite` 119 passed / 1 ignored, catalyst suites 93 passed on the current HEAD.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4068594144
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3227,284 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // Checks the result, and that every node of the optimized plan can produce the attributes
+ // it references. The latter is what this fix is about: an existence join whose condition
+ // references an attribute produced by neither of its children can still return the right
+ // answer when the nested relation is empty at runtime, because the invalid condition is
+ // then never bound. checkAnswer alone would not catch it, as the missing input checks it
+ // runs only look at the root of the plan.
+ def checkAnswerAndPlan(query: String, expected: Seq[Row]): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ checkAnswer(df, expected)
+ }
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // would drop its correlated predicate col1 = c1 and return an extra row, so it is rejected.
Review Comment:
**Nit (P3):** This fixture does not actually return an extra row when `col1 = c1` is dropped. For every `a` present in `t2`, the existing `a = c1` disjunct already makes that value match; for every absent `a`, dropping the nested correlation can only contribute other `c1` values. With these rows both forms return 2 and 3. Please either choose distinguishing data/predicates or remove the result-change claim.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +158,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
Review Comment:
**Blocking (P1):** The EXISTS arms classify ownership against the raw `sub`, while both IN arms call `dedupSubqueryOnSelfJoin` first. For a self-correlated EXISTS with colliding ExprIds, `effectivelyReferencesPlan` can therefore see the same nested expression on both sides and leave it for the both-plan rejection, even though the established deduplication would separate the attributes. Please deduplicate before routing here and in the NOT EXISTS arm, or otherwise ensure routing uses the deduplicated subplan.
**Recommended change:** Normalize ExprId conflicts for EXISTS/NOT EXISTS before attribute-owner routing, keep the routed condition synchronized with aliases, and cover nested subqueries in self-correlated EXISTS and NOT EXISTS.
**Why this works:** Introduce a single preparation step for each predicate-subquery arm that produces an unambiguous outer/subplan pair and corresponding expressions before effective-reference classification. Reuse the prepared plan through routing and final join construction rather than deduplicating after classification.
**Scope:** Make self-join deduplication and nested-subquery routing order consistent across all four predicate-subquery forms.
**Compatibility:** IN and NOT IN keep their existing pre-routing deduplication semantics and all four arms retain correct correlated predicates.
**Risks:** Moving deduplication without rewriting correlated condition references can create ambiguous or stale join expressions.
**Constraints:** Preserve existing SPARK-21835 self-join semantics and the genuine both-plans rejection.
**Success:** A self-correlated EXISTS nested subquery that semantically references one plan is routed to that plan despite pre-dedup ExprId overlap. A genuinely both-side correlated nested subquery is still rejected.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,206 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) && !effectivelyReferencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
Review Comment:
**Nit (P3):** `NOT` is not a context where UNKNOWN and false are indistinguishable: `NOT false` is true, while `NOT UNKNOWN` remains UNKNOWN. The direct NOT IN case is handled safely for a different reason (its null-aware join rewrite). Please narrow this contract to the contexts where only truth is observed and explain the separate NOT IN handling.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,206 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) && !effectivelyReferencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ case Not(child) => unknownSensitiveInSubqueries(child, inPredicate)
+ case in: InSubquery =>
+ if (!inPredicate && inSubqueryMayBeUnknown(in)) Seq(in) else Nil
+ // The join condition of a sub-query expression is a predicate, evaluated by the join that
+ // the sub-query is rewritten into.
+ case sq: SubqueryExpression =>
+ sq.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ case other =>
+ other.children.flatMap(unknownSensitiveInSubqueries(_, inPredicate = false))
+ }
+
+ /**
+ * Returns true if `e` is a positive IN sub-query whose row comparison can evaluate to unknown,
+ * that is one that can return NULL rather than only TRUE or FALSE. An existence join cannot
+ * represent that third value, see [[rewriteExistentialExprInJoinCondition]]. NOT IN is excluded,
+ * as [[rewriteExistentialExprWithAttrs]] gives it a null-aware join condition of its own.
+ */
+ private def inSubqueryMayBeUnknown(e: Expression): Boolean = e match {
+ case InSubquery(values, listQuery) =>
+ values.zip(listQuery.plan.output).exists { case (v, o) => v.nullable || o.nullable }
Review Comment:
**Non-blocking (P2):** The new tests make only the list-query output nullable, so they do not protect the `v.nullable` half of this guard. A nullable compared value with a non-nullable, nonmatching right side can produce the same observable UNKNOWN. Please add a case for that independent source; otherwise removing `v.nullable ||` leaves all of the new tests green.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,14 +405,26 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
+ def rewrite(expr: Expression): Expression = {
Review Comment:
**Non-blocking (P2):** This hand-written traversal no longer provides the metadata handling that `transformDownWithPruning` did: replacements were created under the original node's `CurrentOrigin` and copied its TreeNode tags. The new attributes and rewritten subquery expressions are returned directly, so diagnostics can inherit the enclosing filter's origin and tags can be lost. Please preserve the replaced node's origin and tags around each replacement, or retain equivalent TreeNode transform semantics.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,206 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlan(sq, outerPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, inPredicate = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlan(sq, outerPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) && !effectivelyReferencesPlan(e, outerPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and is not consumed by a
+ * predicate, so that rewriting them into an existence join, whose `exists` attribute is FALSE
+ * where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `inPredicate` states whether the value of `expr` is only ever tested for being TRUE, which
+ * holds for the operands of AND, OR and NOT within a condition that ends up in a Filter or a
+ * join condition. Unknown and FALSE cannot be told apart there.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ inPredicate: Boolean): Seq[Expression] = expr match {
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ case Or(left, right) =>
+ unknownSensitiveInSubqueries(left, inPredicate) ++
+ unknownSensitiveInSubqueries(right, inPredicate)
+ // NOT IN is rewritten with a null-aware join condition, which is two-valued as well.
+ case Not(in: InSubquery) =>
Review Comment:
**Non-blocking (P2):** When this branch is rejected, `Seq(in)` discards the `Not`, so `nestedInSubqueryWithUnknownResultError` renders the user's NOT IN expression as IN. Preserve the full `Not(in)` for the diagnostic. Please also assert `subqueryExpression` for this error and both parameters of the new both-plans error; the current condition-only checks cannot catch this regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4072209586
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -414,7 +433,8 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
existenceJoin, newCondition, subHint)
introducedAttrs += exists
exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
val exists = AttributeReference("exists", BooleanType, nullable = false)()
// Deduplicate conflicting attributes if any.
val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
Review Comment:
You are right about the asymmetry, and about its effect. I could not reach it from SQL, because
`DeduplicateRelations` removes the overlap from every analyzed query I tried, so I built the state
directly as a plan in `RewriteSubquerySuite`, with `a` as the same attribute in the outer and in the
subquery plan. The nested subquery, which references only the subquery it is nested in, is then
rejected with `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY` — the misdescribed rejection you
predicted.
I did not deduplicate `sub` before the call, though, because for the EXISTS arms that is not
equivalent to what the IN arms do. `dedupSubqueryOnSelfJoin` aliases the subquery plan's `a` to a
fresh ExprId, while the hoisted `conditions` still say `a`; the semi join's left side is the outer
plan, which still produces `a`, so the condition would rebind to the outer attribute — a correlated
predicate silently turned into a self-reference, with no error. The IN arms are safe from that only
because they rebuild `inConditions` from `dedupSub.output`; the hoisted `conditions` they pass
through have the same exposure.
What I changed instead is the classification, so that it behaves as though the subplan had been
deduplicated. `effectivelyReferencesPlanOnly` attributes an attribute that *both* plans produce to
the subquery plan, on the grounds that a nested subquery resolves in the scope of the subquery it
sits in, so a shared ExprId is that subquery's own. Only an attribute the outer plan alone produces
now counts as an outer reference, which is what the IN arms effectively get from their deduplicated
plan.
One thing that came out of testing it, relevant to your point (b) in the dedup-anchor comment: in
that state the rewrite cannot finish either way. `buildJoin` hands the semi join condition to
`dedupSubqueryOnSelfJoin`, which has always rejected a condition naming a duplicated attribute, so
the query ends at `_LEGACY_ERROR_TEMP_1212`. A correlated EXISTS's hoisted condition always names
subplan attributes, so any shared ExprId it mentions lands there. The routing fix therefore changes
which error the user sees — from a 0A000 that misdescribes the query to the long-standing conflict
error that names the duplicated attribute — rather than making the query succeed. The new test
asserts the conflict error and fails with the 0A000 if the classification is reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4072311118
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +158,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
Review Comment:
You are right about the asymmetry, and about its effect. I could not reach it from SQL, because
`DeduplicateRelations` removes the overlap from every analyzed query I tried, so I built the state
directly as a plan in `RewriteSubquerySuite`, with `a` as the same attribute in the outer and in the
subquery plan. The nested subquery, which references only the subquery it is nested in, is then
rejected with `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY` — the misdescribed rejection you
predicted.
I did not deduplicate `sub` before the call, though, because for the EXISTS arms that is not
equivalent to what the IN arms do. `dedupSubqueryOnSelfJoin` aliases the subquery plan's `a` to a
fresh ExprId, while the hoisted `conditions` still say `a`; the semi join's left side is the outer
plan, which still produces `a`, so the condition would rebind to the outer attribute — a correlated
predicate silently turned into a self-reference, with no error. The IN arms are safe from that only
because they rebuild `inConditions` from `dedupSub.output`; the hoisted `conditions` they pass
through have the same exposure.
What I changed instead is the classification, so that it behaves as though the subplan had been
deduplicated. `effectivelyReferencesPlanOnly` attributes an attribute that *both* plans produce to
the subquery plan, on the grounds that a nested subquery resolves in the scope of the subquery it
sits in, so a shared ExprId is that subquery's own. Only an attribute the outer plan alone produces
now counts as an outer reference, which is what the IN arms effectively get from their deduplicated
plan.
One thing that came out of testing it, relevant to your point (b) in the dedup-anchor comment: in
that state the rewrite cannot finish either way. `buildJoin` hands the semi join condition to
`dedupSubqueryOnSelfJoin`, which has always rejected a condition naming a duplicated attribute, so
the query ends at `_LEGACY_ERROR_TEMP_1212`. A correlated EXISTS's hoisted condition always names
subplan attributes, so any shared ExprId it mentions lands there. The routing fix therefore changes
which error the user sees — from a 0A000 that misdescribes the query to the long-standing conflict
error that names the duplicated attribute — rather than making the query succeed. The new test
asserts the conflict error and fails with the 0A000 if the classification is reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4346080173
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/statsEstimation/JoinEstimationSuite.scala:
##########
@@ -466,6 +466,51 @@ class JoinEstimationSuite extends StatsEstimationTestBase {
}
}
+ test("existence join") {
+ // table2 (key-1-2 int, key-2-4 int): (1, 2), (2, 3), (2, 4)
+ // table3 (key-1-2 int, key-2-3 int): (1, 2), (2, 3)
+ // An existence join returns every row of its left child with one extra Boolean attribute, so
+ // its cardinality is the left child's and its width is the left child's plus that attribute.
+ // The right child's cardinality must not enter either, as the generic product estimate would
+ // have it, since that can disqualify a broadcast of an otherwise small build side.
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val join = Join(table2, table3, ExistenceJoin(exists),
+ Some(EqualTo(nameToAttr("key-2-4"), nameToAttr("key-2-3"))), JoinHint.NONE)
+
+ val expectedStats = Statistics(
+ // Three rows of the left child, each widened by the Boolean, whose default size is 1.
+ sizeInBytes = 3 * (8 + 4 * 2 + 1),
+ rowCount = Some(3),
+ // Only the left child's column stats: the right child's describe attributes that an
+ // existence join does not output.
+ attributeStats = AttributeMap(Seq(nameToColInfo("key-1-2"), nameToColInfo("key-2-4"))))
+ assert(join.stats == expectedStats)
+ }
+
+ test("existence join with an empty or unestimated right child") {
+ // Only the left child's row count enters the estimate, so a right child with no row count,
Review Comment:
Confirmed: NoRowCountPlan omits the row count and the renamed test now pins the left-only precondition.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4281437506","thread_id":"inline:4281437506","verdict_sha256":"3bb54f2a3a4b20553e9d3ca5caee9c4a8fc61921fc05f7c2363e10d0f4e5e7b0"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,481 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. A nested sub-query resolves in the scope
+ * of the sub-query it is nested in, so an attribute both plans produce is one of that
+ * sub-query's own; attributing it to `outerPlan` instead would let a single shared attribute
+ * satisfy both sides of the classification and have a nested sub-query referencing only the
+ * inner query rejected as referencing both.
+ *
+ * No plan the analyzer produces from SQL reaches this: `DeduplicateRelations` has given the
+ * sub-query its own ExprIds, and the EXISTS arms classify against the raw sub-query plan while
+ * the IN arms classify against the plan `dedupSubqueryOnSelfJoin` has already separated from the
+ * outer plan. What does reach it is the shared-attribute fixture in `RewriteSubquerySuite`,
+ * which builds the overlap directly because the analyzer is not involved there. That fixture
+ * pins the shape, not the outcome of this ordering: `dedupSubqueryOnSelfJoin`'s pre-existing
+ * conflict check reports the duplicated attribute first, so it ends in the same error whichever
+ * plan the shared attribute is attributed to.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite
+ * without rejecting the sub-query, as a sub-query referencing neither plan can be left in the
+ * join condition unharmed.
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(sq)) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(ownedBySubPlan)
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) =
+ rewriteExistentialExprWithAttrs(conditions, subPlan, ownedBySubPlan)
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ // COALESCE(x, false) maps both unknown and FALSE to FALSE, so only truth is observed. A
+ // different default does not: COALESCE(x, true) is TRUE where x is unknown and FALSE where x
+ // is FALSE.
+ case Coalesce(child +: rest) if rest.headOption.contains(Literal.FalseLiteral) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
Review Comment:
Confirmed: the comment now says the NOT IN is reported in a value position and descended into otherwise, matching collectOrDescend.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4281437515","thread_id":"inline:4281437515","verdict_sha256":"3bb54f2a3a4b20553e9d3ca5caee9c4a8fc61921fc05f7c2363e10d0f4e5e7b0"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,481 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. A nested sub-query resolves in the scope
+ * of the sub-query it is nested in, so an attribute both plans produce is one of that
+ * sub-query's own; attributing it to `outerPlan` instead would let a single shared attribute
Review Comment:
Thanks, the two scaladoc paragraphs now agree on the swap reading. The fixture comment at RewriteSubquerySuite.scala:181-183 still describes the 'as well' alternative, so the explanations do not yet pick the same one; see the comment there.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4281437524","thread_id":"inline:4281437524","verdict_sha256":"3bb54f2a3a4b20553e9d3ca5caee9c4a8fc61921fc05f7c2363e10d0f4e5e7b0"} -->
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,540 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: nested IN subquery whose result can be unknown") {
+ withTempView("t1", "t2", "t2n", "t3n", "t3", "t4", "tb", "t2r", "t2rn", "t6", "t7") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // A nullable compared value, against a subquery column that is not nullable.
+ Seq(Some(1), None).toDF("c1").persist().createOrReplaceTempView("t2n")
+ Seq((1, 1), (9, 9)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ // A boolean relation, to compare the result of a nested IN subquery against.
+ Seq(false).toDF("b").persist().createOrReplaceTempView("tb")
+ // Row-valued shapes. t2r and t7 are not nullable anywhere; the nullability of t2rn and of
+ // t6 sits in the second field, so the first pair of the comparison does not expose it.
+ Seq((1, 1), (8, 8), (9, 9)).toDF("c1", "c2").persist().createOrReplaceTempView("t2r")
+ Seq((1, Option.empty[Int]), (8, Some(8))).toDF("c1", "c2n").persist()
+ .createOrReplaceTempView("t2rn")
+ Seq((1, Option.empty[Int]), (5, Some(5))).toDF("col1", "col2").persist()
+ .createOrReplaceTempView("t6")
+ Seq((1, 1), (5, 5)).toDF("col1", "col2").persist().createOrReplaceTempView("t7")
+ // The NULL matches no value of t2, so c1 IN (SELECT col1 FROM t3n) is unknown there, and
+ // c1 NOT IN (SELECT col1 FROM t3n) is never true.
+ Seq(Some(3), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // An existence join only records whether a row matched, so its exists attribute is false
+ // where IN is unknown. That is not observable while the value feeds a predicate, but it is
+ // once it reaches anything else, so these are rejected rather than rewritten. Both
+ // configurations are checked, as the rejection must not depend on the rewrite of predicate
+ // subqueries in join conditions.
+ Seq(
+ "(c1 IN (SELECT col1 FROM t3n)) <=> false",
+ "(c1 IN (SELECT col1 FROM t3n)) IS NULL",
+ // COALESCE to a non-false default does tell unknown from false.
+ "COALESCE(c1 IN (SELECT col1 FROM t3n), true)",
+ "(c1 NOT IN (SELECT col1 FROM t3n)) <=> false").foreach { predicate =>
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT * FROM t1 WHERE EXISTS (SELECT 1 FROM t2 WHERE a = c1 OR $predicate)")
+ .collect()
+ }
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT")
+ // A NOT IN must be reported as a NOT IN, not as the IN underneath it.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.startsWith("(NOT ") == predicate.contains("NOT IN"),
+ s"unexpected subqueryExpression for $predicate: $reported")
+ }
+ }
+ }
+
+ // The ordinary case: the value of the nested IN feeds a predicate, where unknown and
+ // false cannot be told apart, so it is still rewritten even though it can be unknown.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2 WHERE a = c1 OR c1 IN (SELECT col1 FROM t3n))""".stripMargin,
+ Row(1) :: Nil)
+
+ // And a nested IN that cannot be unknown is rewritten even outside a predicate.
+ checkAnswerAndPlan(
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1 OR ((c1 IN (SELECT col1 FROM t3)) <=> false))""".stripMargin,
+ Row(1) :: Row(2) :: Row(3) :: Nil)
+
+ // Only truth is observed by `<=> true`, by an IF predicate and by a CASE WHEN
+ // condition, so those are rewritten rather than rejected. SimplifyConditionals folds
+ // the latter two into the first, so ordinary SQL reaches this position.
+ //
+ // The result branches of a conditional are consumed wherever the conditional is, so
+ // under a filter predicate they are truth-only as well. The last two cases do not
Review Comment:
Confirmed: the comment now names the three result-branch cases and the SimplifyConditionals folding rule.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4281437538","thread_id":"inline:4281437538","verdict_sha256":"3bb54f2a3a4b20553e9d3ca5caee9c4a8fc61921fc05f7c2363e10d0f4e5e7b0"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4356967243
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,568 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is deliberately left where it is. It has no hoisted condition to
+ * lose, and `InSubqueryExec` evaluates it with the three-valued semantics that the `exists`
+ * attribute of an existence join cannot carry.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ // Only a correlated sub-query has to be consumed here, see above.
+ val carried = condition.collect {
+ case sq @ (_: Exists | _: InSubquery) if hasCorrelatedCondition(sq) => sq
+ }
+ if (carried.isEmpty) {
+ return join
+ }
+ // Where the value of a carried IN is observed beyond being TRUE, an existence join cannot
+ // stand in for it, exactly as on the routes above.
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+
+ val rejected = carried.filter { sq =>
+ observesUnknown(sq) ||
+ (referencesInput(sq, join.left) && referencesInput(sq, join.right))
+ }
+ if (rejected.nonEmpty) {
+ // A sub-query that needs both inputs can be rewritten against neither, and one whose
+ // unknown is observable cannot become an `exists` attribute at all.
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(rejected)
Review Comment:
Fixed. `rejected` is split into two sets, each raising the condition whose message describes it:
```scala
val referencingBothInputs = carried.filter { sq =>
referencesBothInputs(sq) &&
(hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
}
if (referencingBothInputs.nonEmpty) { throw ...nestedSubqueryReferencingOuterAndInnerQueryError... }
val unknownResult = carried.filter(sq => referencesOneInput(sq) && observesUnknown(sq))
if (unknownResult.nonEmpty) { throw ...nestedInSubqueryWithUnknownResultError... }
```
Your example now reports `NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT` under both configurations, and the new test asserts the condition for a carried IN rejected for that reason, both correlated and uncorrelated -- so the throw is reached by a test now, which it was not before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6474335717
Gentle Ping! @cloud-fan - Could you please take a look and inform if any other changes are required before we check this in? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4380488016
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothInputs.toSeq)
+ }
+ val unknownResult = carried.filter(sq => referencesOneInput(sq) && observesUnknown(sq))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult.toSeq)
+ }
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ carried.filter(sq => referencesOneInput(sq) && isInput(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) = rewriteExistentialExprWithAttrs(Seq(sq), p)
+ newCondition = newCondition.transform {
+ case e if e.fastEquals(sq) => rewritten.get
+ }
+ newInput
+ }
+ }
+ val newLeft = route(join.left, referencesInput(_, join.left))
+ val newRight = route(join.right, referencesInput(_, join.right))
+ if (newLeft.fastEquals(join.left) && newRight.fastEquals(join.right)) {
+ join
+ } else {
+ // Drop the `exists` attributes the nested existence joins introduced, keeping this join's
+ // own output, which already carries its own.
+ Project(join.output, join.copy(
+ left = newLeft, right = newRight, condition = Some(newCondition)))
+ }
+ }
+
+ /**
+ * Collects the existential sub-queries carried by the condition of an existence join, as the
+ * units [[rewriteExistentialExprWithAttrs]] rewrites: a NOT IN is collected as the [[Not]] over
+ * the [[InSubquery]], so the rewrite takes its null-aware arm and the unknown-result collector,
+ * which reports the same node, can be compared against it.
+ *
+ * A collected sub-query is not descended into. One nested below it is reached when it is
+ * rewritten, since the join built for it goes through the routing in turn, and one below a
+ * sub-query left in place is left with it, as everywhere else in this rule.
+ */
+ private def collectCarriedExistentialExprs(
+ expr: Expression,
+ collected: ArrayBuffer[Expression]): Unit = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return
+ }
+ expr match {
+ case sq @ Not(_: InSubquery) => collected += sq
+ case sq: Exists => collected += sq
+ case sq: InSubquery => collected += sq
+ case other => other.children.foreach(collectCarriedExistentialExprs(_, collected))
+ }
+ }
+
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan = rewriteCarriedExistentialExprs(
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint))
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint))
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(
+ Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint))
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. A nested sub-query resolves in the scope
+ * of the sub-query it is nested in, so an attribute both plans produce is one of that
+ * sub-query's own.
+ *
+ * The probes are exclusive, so swapping their order would attribute such an attribute to
+ * `outerPlan` alone: `subPlanOnly` and `bothPlans` would both be false, and a nested sub-query
+ * whose only reference is the shared attribute would be declined and left in the join condition
+ * rather than routed to the sub-query plan that can evaluate it.
+ *
+ * No plan the analyzer produces from SQL reaches this: `DeduplicateRelations` has given the
+ * sub-query its own ExprIds, and the EXISTS arms classify against the raw sub-query plan while
+ * the IN arms classify against the plan `dedupSubqueryOnSelfJoin` has already separated from the
+ * outer plan. What does reach it is the shared-attribute fixture in `RewriteSubquerySuite`,
+ * which builds the overlap directly because the analyzer is not involved there. That fixture
+ * pins the shape, not the outcome of this ordering: under either order it ends in the same
+ * error, since `dedupSubqueryOnSelfJoin`'s pre-existing conflict check reports the duplicated
+ * attribute that the hoisted condition mentions.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite here.
+ * That does not leave the sub-query unexamined: one carried by a nested sub-query that is
+ * rewritten is routed against the inputs of the join built for it, see
+ * [[rewriteCarriedExistentialExprs]].
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
Review Comment:
Fixed. The scaladoc of `rewriteExistentialExprInJoinCondition` now says the rejection covers the sub-queries the sub-query plan owns, and that an IN that only references the outer plan is still rewritten two-valued, as before this PR, tracked by SPARK-59862.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5681963949
@MaxGekk @LuciferYang Could you please help review this change? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5731841503
Thanks for the ping. I reviewed the current head and found one test-coverage gap: several new IN/NOT IN fixtures do not make their asserted rows depend on the nested subquery. I will include the concrete case in the review.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"conversation:5729170954","thread_id":"conversation:5729170954","verdict_sha256":"79c14807de6bf901c8e58d252b5f36d4635f46f9b33f2af559fb7e983c81d7df"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4040745402
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -454,6 +470,70 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
}
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
Review Comment:
Thanks for noting this, the two lines in the `case j: Join` handler now call `referencesPlan`, so the classification
of a sub-query by the side it references exists once. Also thanks for the note on
`PredicateHelper.canEvaluate`; it uses `subsetOf` rather than a non-empty intersection, so it is a
different test and not usable for either site.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4148332391
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) => collectOrDescend(not, in, truthOnly, reported)
+ case Not(child) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = false, reported)
+ case in: InSubquery => collectOrDescend(in, in, truthOnly, reported)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = true, reported))
+ sq.getOuterAttrs.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ case other =>
+ other.children.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ }
+
+ /**
+ * Appends `report` to `reported` if the IN sub-query `in` would lose an observable unknown, and
+ * otherwise descends into the values it compares and into its list query. The descent matters
+ * because [[InSubquery]] is a [[Predicate]] rather than a [[SubqueryExpression]], so an IN
+ * sub-query nested in the values or in the hoisted join condition of another one is only reached
+ * from here.
+ */
+ private def collectOrDescend(
+ report: Expression,
+ in: InSubquery,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = {
+ if (!truthOnly && inSubqueryMayBeUnknown(in)) {
+ reported += report
+ } else {
+ in.values.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ collectUnknownSensitiveInSubqueries(in.query, truthOnly, reported)
+ }
+ }
+
+ /**
+ * Returns true if the row comparison of the IN sub-query `in` can evaluate to unknown, that is
+ * if it can return NULL rather than only TRUE or FALSE. An existence join cannot represent that
+ * third value, see [[rewriteExistentialExprInJoinCondition]].
+ *
+ * This deliberately does not use `InSubquery.nullable`, which under
+ * `spark.sql.legacy.inSubqueryNullability` (SPARK-43413) reports only the nullability of the
+ * compared values and ignores that of the list query's columns. What is protected here is the
+ * three-valued result the rewrite would drop, which does not depend on that configuration, so
+ * the nullability is re-derived from both sides.
+ */
+ private def inSubqueryMayBeUnknown(in: InSubquery): Boolean = {
+ in.values.zip(in.query.childOutputs).exists { case (v, o) => v.nullable || o.nullable }
+ }
+
+ /**
+ * Returns true if `e` is an existential sub-query expression that still carries a correlated
+ * condition, that is one that [[PullupCorrelatedPredicates]] hoisted into it and that would be
+ * lost were the sub-query left in a join condition.
+ *
+ * This is decided from that condition rather than from `isCorrelated`, which reports the outer
Review Comment:
Fixed. The scaladoc now says `isCorrelated` is `outerAttrs.nonEmpty`, which is what makes it the
wrong question here, rather than describing it as the attribute collection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4148327693
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
+ // are examined. Under any other NOT, unknown is observable and `truthOnly` does not carry.
+ case not @ Not(in: InSubquery) => collectOrDescend(not, in, truthOnly, reported)
+ case Not(child) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = false, reported)
+ case in: InSubquery => collectOrDescend(in, in, truthOnly, reported)
+ case sq: SubqueryExpression =>
+ // Only the hoisted join condition of a sub-query expression is a predicate, evaluated by
+ // the join it is rewritten into. Its outer attributes are value expressions.
+ sq.getJoinCond.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = true, reported))
+ sq.getOuterAttrs.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ case other =>
+ other.children.foreach(
+ collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ }
+
+ /**
+ * Appends `report` to `reported` if the IN sub-query `in` would lose an observable unknown, and
+ * otherwise descends into the values it compares and into its list query. The descent matters
+ * because [[InSubquery]] is a [[Predicate]] rather than a [[SubqueryExpression]], so an IN
+ * sub-query nested in the values or in the hoisted join condition of another one is only reached
+ * from here.
+ */
+ private def collectOrDescend(
+ report: Expression,
+ in: InSubquery,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = {
+ if (!truthOnly && inSubqueryMayBeUnknown(in)) {
+ reported += report
+ } else {
+ in.values.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly = false, reported))
+ collectUnknownSensitiveInSubqueries(in.query, truthOnly, reported)
+ }
+ }
+
+ /**
+ * Returns true if the row comparison of the IN sub-query `in` can evaluate to unknown, that is
+ * if it can return NULL rather than only TRUE or FALSE. An existence join cannot represent that
+ * third value, see [[rewriteExistentialExprInJoinCondition]].
+ *
+ * This deliberately does not use `InSubquery.nullable`, which under
+ * `spark.sql.legacy.inSubqueryNullability` (SPARK-43413) reports only the nullability of the
+ * compared values and ignores that of the list query's columns. What is protected here is the
+ * three-valued result the rewrite would drop, which does not depend on that configuration, so
+ * the nullability is re-derived from both sides.
+ */
+ private def inSubqueryMayBeUnknown(in: InSubquery): Boolean = {
+ in.values.zip(in.query.childOutputs).exists { case (v, o) => v.nullable || o.nullable }
Review Comment:
Fixed: the nullability check now zips iterators, so a wide row-valued IN stops at the first nullable
side instead of pairing the whole arity first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4148339464
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,393 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
Review Comment:
Fixed. The scaladoc now names both: the subquery survives that rewrite when
`decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and also when it is an
uncorrelated IN and `optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled, since the
relevant-subquery filter then does not consider it at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4064435485
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) && referencesPlan(sq, subPlan) &&
Review Comment:
You are right, and it was worse than a spurious rejection: it was a regression against released
Spark. Your query works on 4.0.1, returning `1`:
```sql
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2 WHERE a = c1 OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1));
```
```
-- 4.0.1 optimized plan: the existence join sits on the outer plan, which is valid, because the
-- only attribute the nested subquery still reads is a
Project [a#18]
+- Join LeftSemi, ((a#18 = c1#19) OR exists#22)
:- Join ExistenceJoin(exists#22), (a#18 = col1#20)
: :- LocalRelation [a#18]
: +- LocalRelation <empty>, [col1#20]
+- Project [c1#19]
```
This PR rejected it with `0A000`, so a working query would have started failing.
Classifying correlation from the surviving condition, as you asked, was necessary but not
sufficient on its own: with that alone the subquery was no longer rejected by the new guard, but it
was still classified as referencing the subquery plan, because `references` carries the retained
`c1`. It therefore stayed in the join condition and the pre-existing rewrite of predicate
subqueries in join conditions rejected it under the default configuration instead, with
`UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION`. I only noticed because I checked the query
against 4.0.1 rather than against this PR alone.
So the routing uses the surviving state too:
* `hasCorrelatedCondition` replaces `isCorrelated`, and reads the hoisted join condition, which is
what would actually be lost if the subquery were left in a join condition. `rewriteDomainJoins`
already reasons this way -- its comment notes that BooleanSimplification eliminating every
correlated predicate leaves `joinCond` as `None` and hence no domain join behind;
* `effectiveReferences` gives the attributes of the compared values and of that hoisted condition,
excluding the outer attributes the pull-up retains for idempotency, and the routing and both
guards use it.
Your query is now rewritten against the outer plan as it is on 4.0.1, under both settings of
`decorrelatePredicateSubqueriesInJoinPredicate.enabled`, and the genuinely correlated shape is
still rejected under both:
| | default conf | conf off |
|---|---|---|
| `false AND col1 = c1` (eliminated) | `1` | `1` |
| `col1 = c1` (genuine) | `0A000` | `0A000` |
Added to the regression test, with a comment on why the subquery keeps `c1` while no longer reading
it. Reverting either half of the classification fails that test, reporting
`The subquery (t1.a IN (listquery(t2.c1))) ... references both the outer query and the subquery it
is nested in`, where `listquery(t2.c1)` is the retained attribute.
Note that `referencesPlan` and `effectivelyReferencesPlan` now both exist: the first is the plain
`references` test, still shared with the `case j: Join` handler, and the second is used by the
nested subquery routing. They are deliberately different questions -- whether an expression names
an attribute at all, and whether it can still read one -- so I kept both rather than changing the
pre-existing handler's classification, which is outside this JIRA. Happy to align that one too if
you would prefer it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4064442984
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
Review Comment:
Fixed, the scaladoc now names`spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5764348512
Thanks @cloud-fan for your comments. Also thank you for for settling three-valued-IN issue. I will file the follow-up for the released outer and projection routes,
where the same substitution is already wrong without any correlation involved and will link it here once it is created.
I have addressed all your comments. PTAL!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4133813605
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,372 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, which allocates nothing.
+ // AttributeSet.intersect materialises a set from its argument, so the operand order there
+ // would decide whether this walks the references or the whole output of `plan`.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references an attribute that `plan` produces and `sharedWith`
+ * does not, see [[effectiveReferences]].
+ *
+ * An attribute that both plans produce is attributed to `sharedWith` rather than to `plan`.
+ * This matters when the outer and the sub-query plan share ExprIds, which a self-join can leave
+ * for the optimizer to deduplicate (SPARK-21835): the EXISTS arms classify against the raw
+ * sub-query plan, while the IN arms classify against the plan that
+ * `dedupSubqueryOnSelfJoin` has already separated from the outer plan. Without this a single
+ * shared attribute would satisfy both sides of the classification, so a nested sub-query that
+ * references only the sub-query it is nested in would be taken for one referencing both plans
+ * and rejected. A nested sub-query resolves in the scope of that sub-query, so a shared
+ * attribute is one of its own.
+ */
+ private def effectivelyReferencesPlanOnly(
+ e: Expression,
+ plan: LogicalPlan,
+ sharedWith: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists { attr =>
+ plan.outputSet.contains(attr) && !sharedWith.outputSet.contains(attr)
+ }
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) => AttributeSet(joinCond.flatMap(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet(values.flatMap(_.references) ++ joinCond.flatMap(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // Both collectors below match only Exists and InSubquery, and neither prunes on tree
+ // patterns, so skip them when the hoisted conditions hold no sub-query at all, which is the
+ // common case for a correlated EXISTS or IN.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. Note
+ // that this walks the whole expression, including the join condition of a sub-query that the
+ // rewrite below declines to descend into. That is deliberate: a correlated sub-query hidden
+ // under a declined one would equally be planned without its join condition, or reach
+ // execution unevaluable, so it must be rejected even though nothing would have rewritten it.
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if hasCorrelatedCondition(sq) && effectivelyReferencesPlan(sq, subPlan) &&
+ effectivelyReferencesPlanOnly(sq, outerPlan, subPlan) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(sq => effectivelyReferencesPlan(sq, subPlan) &&
+ !effectivelyReferencesPlanOnly(sq, outerPlan, subPlan))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
+ e => effectivelyReferencesPlan(e, subPlan) &&
+ !effectivelyReferencesPlanOnly(e, outerPlan, subPlan))
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, the predicate of an `If` and the conditions of a `CaseWhen`.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) => Nil
+ case And(left, right) =>
+ unknownSensitiveInSubqueries(left, truthOnly) ++
Review Comment:
Corrected, and thank you for reading the implementation rather than my comment.
`InSubquery.nullable` under `spark.sql.legacy.inSubqueryNullability` is
`values.exists(_.nullable)`, so a nullable compared value stays nullable and only the list query's
column nullability is dropped. The scaladoc now says exactly that, and no longer claims the flag
returns false for either side. The typo is gone.
The nullable-left case is now asserted under both settings of the flag, since the guard re-derives
both sides and that case does not depend on it: a nullable compared value against a non-nullable,
non-matching list query is rejected with
`NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT` whether the flag is on or off, and the test comment states
which half of the nullability the flag actually affects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114373943
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3227,262 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // Checks the result, and that every node of the optimized plan can produce the attributes
+ // it references. The latter is what this fix is about: an existence join whose condition
+ // references an attribute produced by neither of its children can still return the right
+ // answer when the nested relation is empty at runtime, because the invalid condition is
+ // then never bound. checkAnswer alone would not catch it, as the missing input checks it
+ // runs only look at the root of the plan.
+ def checkAnswerAndPlan(query: String, expected: Seq[Row]): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ checkAnswer(df, expected)
+ }
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // would drop its correlated predicate col1 = c1 and return an extra row, so it is rejected.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ val df = sql(uncorrelated)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
Review Comment:
All three points taken. `checkAnswerAndPlan` is now a suite-level `private def` used by all three
tests, and it also asserts plan shape: a semi/anti join exists, an existence join exists, and every
existence join sits under the right side of the semi/anti join, so a rewrite that placed it on the
outer side would now fail. The scaladoc records why `missingInput` cannot see a surviving sub-query
expression — `QueryPlan.references` folds only its outer attributes and `collect` does not walk its
plan — so it is not read as sufficient. The `uncorrelated` case, where the sub-query deliberately
survives in the condition, opts out of the shape check explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114428367
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,14 +405,26 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
+ def rewrite(expr: Expression): Expression = {
Review Comment:
Agreed, and fixed. The traversal was replaced in an earlier round to stop it descending into a
subquery that `canRewrite` declines, and in doing so it lost both things you list, which
`transformDownWithPruning` provided: the rule was applied inside `CurrentOrigin.withOrigin(origin)`,
and `copyTagsFrom` ran when a node was replaced.
Each of the three arms now produces its replacement through a `replacing(sq) { ... }` helper that
wraps the body in `CurrentOrigin.withOrigin(sq.origin)` and copies the replaced expression's tags,
so the `exists` attribute, the synthesized join conditions and the `Join` node are all built under
the origin of the subquery expression they come from. The do-not-descend behaviour and the pruning
check are unaffected.
The most visible consequence was the `SQLQueryContext` of a runtime error raised from a rewritten
join condition, which had started pointing at the enclosing operator instead of the subquery
fragment; that is restored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4056570195
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3227,198 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n") {
+ Seq((1), (2), (3), (7)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+
+ // Checks the result, and that every node of the optimized plan can produce the attributes
+ // it references. The latter is what this fix is about: an existence join whose condition
+ // references an attribute produced by neither of its children can still return the right
+ // answer when the nested relation is empty at runtime, because the invalid condition is
+ // then never bound. checkAnswer alone would not catch it, as the missing input checks it
+ // runs only look at the root of the plan.
+ def checkAnswerAndPlan(query: String, expected: Seq[Row]): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ checkAnswer(df, expected)
+ }
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // Same, with a nested subquery that returns no matching row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(1) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR EXISTS (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is
+ // never true, so only the correlated predicate can be satisfied.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Nil)
+
+ // Without the NULL, c1 NOT IN (3, 9) holds for c1 = 1.
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested subquery that is itself correlated to the query it is nested in.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(1) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // would drop its correlated predicate col1 = c1 and return an extra row, so it is rejected.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ // The rejection must not depend on whether predicate subqueries in join conditions are
+ // decorrelated, as that rewrite rejects the same shape on its own.
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
Review Comment:
Deliberate, and the comment now says so, close to your wording: a correlated subquery hidden under a
declined one would equally be planned without its join condition or reach execution unevaluable, so
it must be rejected even though nothing would have rewritten it. Agreed that narrowing the scan to
match the rewrite is the worse direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4056568739
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3227,198 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n") {
+ Seq((1), (2), (3), (7)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+
+ // Checks the result, and that every node of the optimized plan can produce the attributes
+ // it references. The latter is what this fix is about: an existence join whose condition
+ // references an attribute produced by neither of its children can still return the right
+ // answer when the nested relation is empty at runtime, because the invalid condition is
+ // then never bound. checkAnswer alone would not catch it, as the missing input checks it
+ // runs only look at the root of the plan.
+ def checkAnswerAndPlan(query: String, expected: Seq[Row]): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ checkAnswer(df, expected)
+ }
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // Same, with a nested subquery that returns no matching row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(1) :: Nil)
Review Comment:
Right on all three, including the reduction: with `a = c1` hoisted,
`a IN {c1 ∈ t2 : a = c1 OR nested(c1)}` is `a ∈ t2` whatever `nested` returns. And query8's
`c1 IN (SELECT col1 FROM t3 WHERE col1 = c1)` is indeed equivalent to the uncorrelated form, so it
pinned nothing about correlated nesting.
Rebuilt with your suggestion, `a > c1` as the hoisted disjunct, plus data chosen so each answer
moves if the nested subquery is mishandled. `t1` is now `(1, 2, 3, 7, 9)`:
| query | shape | answer | what breaks it |
|---|---|---|---|
| query2 | EXISTS, nested IN over `t5 = (3, 7)`, disjoint from `t2` | `1, 9` | nested IN wrongly true -> every row |
| query4 | outer IN, `a > c1`, nested IN over `t3` | `9` | nested IN dropped -> empty |
| query5 | outer NOT IN, `a > c1`, nested EXISTS | `1, 2, 3, 7` | nested EXISTS dropped -> `9` comes back |
| query6/7 | nested NOT IN over a relation with/without NULL | `1, 9` / all rows | null-awareness lost |
| query8 | EXISTS, `a > c1`, nested IN over `t4(col1, k)` correlated on `k`, which it does not project | `2, 3, 7, 9` | correlation dropped -> `1` comes back |
`t4 = ((1, 9), (9, 1))` makes the correlated nested IN false for every `c1` in `t2` while the same
IN without its correlated predicate is true for 1 and 9, so correlated and uncorrelated nesting now
give different answers. Every query goes through the tree-wide `missingInput` helper as well.
Your note that query5 never had the `missingInput` half is accurate — `SubqueryExpression.references`
excludes the join condition, so the nested `Exists` has `outerAttrs = {c1}` from the right child and
its pre-fix failure is an `Unevaluable` `Exists` reaching evaluation. With the new data the NOT IN
branch no longer rests on "it does not throw": its answer changes if the nested EXISTS is mishandled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4278473903
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +159,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftSemi, joinCond, subHint)
Project(p.output, join)
case (p, Not(Exists(sub, _, _, conditions, subHint))) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftAnti, joinCond, subHint)
Project(p.output, join)
case (p, InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val (joinCond, outerPlan) = rewriteExistentialExpr(inConditions ++ conditions, p)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(inConditions ++ conditions, p, dedupSub)
Review Comment:
Done, in a separate commit on this branch so the routing fix and the estimation change stay separately reviewable.
### Both statistics modes
`SizeInBytesOnlyStatsPlanVisitor.visitJoin` gains an `ExistenceJoin` arm that rescales the left child's size the same way `visitUnaryNode` does, since the row count is unchanged and only the appended Boolean widens each row:
```scala
case ExistenceJoin(_) =>
visitExistenceJoin(p)
```
`JoinEstimation` gains `estimateExistenceJoin()`, which takes the row count from the left child and sizes the output from `join.output` — that is the left output plus `exists`, so the appended width is accounted for, and the attribute has no column statistics so `getSizePerRow` falls back to its type's default size. Only the left child's attribute statistics are propagated; the right child's describe attributes an existence join does not output.
It requires only `rowCountsExist(join.left)`, so a right child with no statistics, or an empty one, is still estimated rather than falling back to the product.
### Measured
On the shape from this thread, with `spark.sql.cbo.enabled` either way:
| | left | right | node estimate |
|---|---|---|---|
| before | 2,579 | 2,591 | **6,682,189** |
| after | 2,579 | 2,591 | **2,601** |
The old estimate is the product exactly — a 2,591x over-estimate, the right child's size used as a multiplier — for an operator that emits precisely the left child's 2,579 bytes plus one byte per row.
### A TPC-DS plan actually improves
`q10.sf100` under `TPCDSV1_4_PlanStabilityWithStatsSuite` moves, and it is the regression you predicted, in an approved plan. Net operator change:
```
-1 SortMergeJoin +1 BroadcastHashJoin
-2 Exchange +1 BroadcastExchange
-2 Sort
-3 WholeStageCodegen
```
The subtree carrying `Filter [exists,exists]` is now small enough to broadcast, so the shuffle and sort of the customer-side input disappear:
```
- SortMergeJoin [c_current_cdemo_sk,cd_demo_sk]
- ... Sort [c_current_cdemo_sk]
- ... Exchange [c_current_cdemo_sk] #2
+ BroadcastHashJoin [c_current_cdemo_sk,cd_demo_sk]
+ ... BroadcastExchange #2
```
Golden files regenerated with `SPARK_GENERATE_GOLDEN_FILES=1`, not edited. This is the only one of the 8 approved plans containing an `ExistenceJoin` that changes, and only in the CBO-enabled suite — which fits, since that is the mode where `JoinEstimation` used to return `None`.
### Regression
The new plan-selection test in `SubquerySuite` is the one you asked for: a nested inner-routed existence join whose left child is below the broadcast threshold while the product would exceed it, asserting the enclosing equi semi join keeps its broadcast. It is marked `DisableAdaptiveExecution`, since with AQE on runtime statistics recover the plan and the estimate under test is not the one the planner uses. It also notes why the hoisted predicate is `a > c1` rather than the key equality: with a disjunction as the whole condition the join is a broadcast nested loop join whatever the statistics say, so it could not show the effect.
Checked that it discriminates — with the two new arms reverted it fails with
```
6682189 was not less than 5158 the existence join is estimated at 6682189 from a
left child of 2579 and a right child of 2591
```
Statistics regressions cover both modes: `existence join` and `existence join with an empty or unestimated right child` in `JoinEstimationSuite` pin the CBO row count, size and left-only attribute stats, including a right child with a zero row count and one with no column statistics; `existence join is estimated from its left child in both statistics modes` in `BasicStatsEstimationSuite` pins the same invariant through `checkStats` with CBO on and off, and asserts the estimate is not the product.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4369134655
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothInputs.toSeq)
+ }
+ val unknownResult = carried.filter(sq => referencesOneInput(sq) && observesUnknown(sq))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult.toSeq)
+ }
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ carried.filter(sq => referencesOneInput(sq) && isInput(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) = rewriteExistentialExprWithAttrs(Seq(sq), p)
+ newCondition = newCondition.transform {
Review Comment:
**Blocking (P1):** When one IN sub-query instance is carried both bare and under NOT, this substitution breaks the NOT IN. `collectCarriedExistentialExprs` returns the bare IN and the `Not(IN)` as two units, bare first. Routing the bare unit rewrites every `fastEquals` occurrence of the IN, including the one inside the `Not`. The `Not(IN)` unit then builds its null-aware existence join, nothing references it, and the condition keeps `NOT exists` over a two-valued match.
Filter pushdown produces this from ordinary SQL, because alias substitution puts the same IN instance in both positions. With t1(a) = 1, -1, NULL; t2(c1) = 1; t3(col1) = 3, NULL:
```sql
SELECT * FROM (SELECT a, a IN (SELECT col1 FROM t3) AS b FROM t1) s
WHERE s.a > 0 OR EXISTS (SELECT 1 FROM t2 WHERE s.a = t2.c1 OR s.b OR NOT s.b)
```
Standard SQL keeps only a = 1. I ran it on this head: it returns a = -1, 1 and NULL under both values of `decorrelatePredicateSubqueriesInJoinPredicate.enabled`. The optimized condition is `((a = c1) OR exists#340) OR NOT exists#340`, next to an unreferenced `ExistenceJoin(exists#341), ((a = col1) OR isnull((a = col1)))`. Written with two distinct IN sub-queries, the same predicate returns only a = 1.
With this file reverted to the merge base, the flag-off configuration returns only a = 1, because the IN stays in the condition as `InSubqueryExec`. With the default, the merge base already returned the same wrong rows through the identical `fastEquals` pattern in the `case j: Join` arm, so that part is not new; the router now also brings it to the flag-off path.
Substituting each unit only where it was collected, so that a bare IN's substitution does not descend into a `Not(IN)` unit, fixes both configurations. Routing the NOT IN units before the bare ones is an equivalent minimal fix.
This extends an [existing discussion](https://github.com/apache/spark/pull/58656#issuecomment-6380316971).
**Verification:**
- **Regression:** With t1(a) = 1, -1, NULL, t2(c1) = 1 and t3(col1) = 3, NULL, `SELECT * FROM (SELECT a, a IN (SELECT col1 FROM t3) AS b FROM t1) s WHERE s.a > 0 OR EXISTS (SELECT 1 FROM t2 WHERE s.a = t2.c1 OR s.b OR NOT s.b)` returns only a = 1 under both values of decorrelatePredicateSubqueriesInJoinPredicate, as the same predicate written with two distinct IN sub-queries does.
- **Compatibility:** The existing carried NOT IN, carried-routing and NOT IN residual cases keep their answers and plan placement.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
Review Comment:
**Nit (P3):** Deleting this throw leaves every test green. Both assertions of `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY` in SubquerySuite (the correlated NOT IN case and `carriedCorrelation`) are raised by the first-level rejection in `rewriteExistentialExprInSubqueryPlan`, and every carried-routing case reads only the existence join's right input.
A shape that reaches it is a third-level `c1 IN (SELECT x FROM t5 WHERE x = k)` inside a nested `c1 IN (SELECT col1 FROM t4 WHERE k = c1 OR ...)`. Its compared value comes from the left input and its hoisted `x = k` reads the right one. Without the throw, the flag-off configuration plans it as an in-subquery filter and drops `x = k`, returning extra rows, while the default fails with `UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION`. A case asserting the condition under both flag values would pin it.
**Verification:**
- **Regression:** A carried correlated sub-query that references both inputs of the existence join built for its parent fails with NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY under both values of decorrelatePredicateSubqueriesInJoinPredicate. The case fails if the carried both-inputs rejection is removed, rather than returning extra rows or a different error class.
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,749 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. The EXISTS form of this query fails on the same condition, raised by the same
+ // rejection in rewriteExistentialExprInSubqueryPlan before any join exists; were that
+ // rejection removed, the EXISTS form would instead reach the join-condition handling, which
+ // rejects it with UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION under the default
+ // configuration, so only the NOT IN form pins this rejection on its own.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(correlated).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ // The message must name the subquery that was rejected.
+ val reported = e.getMessageParameters.get("subqueryExpression")
+ assert(reported.contains("IN") && reported.contains("listquery"),
+ s"unexpected subqueryExpression: $reported")
+ }
+ }
+
+ // A correlated predicate that BooleanSimplification has eliminated:
+ // PullupCorrelatedPredicates retains c1 as an outer attribute of the nested subquery for
+ // idempotency, but the subquery
+ // no longer reads it, so there is no condition to lose. It must still be rewritten against
+ // the outer plan, which is the only plan it effectively references, rather than be treated
+ // as referencing the subquery plan and rejected.
+ val simplifiedAwayCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswer(sql(simplifiedAwayCorrelation), Row(1) :: Nil)
+ }
+ }
+
+ // A subquery that stays in the join condition stays there whole, since the rewrite does
+ // not descend into one it declines, so a correlated subquery below it is left as well.
+ // Its own references make it look routable, being owned by the subquery plan, but
+ // nothing reaches it and planning a list query as an in-subquery filter drops the
+ // predicate the pull-up hoisted into it. The enclosing subquery is rejected for
+ // carrying it: PostgreSQL and DuckDB return 1 and 3 here, and losing the correlation
+ // returns 2 as well.
+ val carriedCorrelation =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
+ | IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ val e = intercept[AnalysisException](sql(carriedCorrelation).collect())
+ assert(e.getCondition == "UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY." +
+ "NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY")
+ }
+ }
+
+ // An uncorrelated nested subquery referencing both queries has no correlated predicate to
+ // lose, so it stays in the join condition and is planned as an in-subquery filter. The
+ // configuration is disabled because the rewrite of predicate subqueries in join conditions
+ // rejects a subquery referencing both join inputs, which is what this one becomes.
+ val uncorrelated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR (a + c1) IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ withSQLConf(
+ SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key -> "false") {
+ // No existence join here: this subquery stays in the join condition, which is the
+ // point of the case, so only the result and the reachability of the plan are
+ // checked.
+ checkAnswerAndPlan(uncorrelated, Row(1) :: Row(2) :: Nil,
+ existenceJoinOnSubqueryPlan = false)
+ }
+ }
+ }
+
+ test("SPARK-59351: existential subquery carried by a rewritten nested subquery") {
+ // The nested IN over t4 is owned by the subquery plan, so it is rewritten into an existence
+ // join and the condition PullupCorrelatedPredicates hoisted into it is grafted onto that
+ // join. That condition carries a third-level IN correlated to t4, which references neither
+ // the outer nor the subquery plan and so is routed by neither of the guards above. It has to
+ // be rewritten against the inputs of the join just built: left in place, PlanSubqueries plans
+ // it as an in-subquery filter and drops its own correlated predicate col1 = k, which returns
+ // every row of t1 instead of the one row where the correlation holds.
+ withTempView("t1", "t2", "t4", "t5") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ // No k equals any c1, so only the third-level IN can satisfy the hoisted disjunction, and
+ // its correlated predicate col1 = k is false for every row of t4: dropping it makes the
+ // level-2 IN hold for c1 = 1 and returns 1, 2 and 3.
+ Seq((1, 100), (9, 7)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(1).toDF("col1").persist().createOrReplaceTempView("t5")
+ val query =
+ """SELECT * FROM t1 WHERE EXISTS (
+ | SELECT 1 FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t4
+ | WHERE k = c1 OR col1 IN (SELECT col1 FROM t5 WHERE col1 = k)))""".stripMargin
+ // The routing must not depend on the rewrite of predicate subqueries in join conditions,
+ // which is what used to make these two configurations disagree: it reached the grafted
+ // condition only when enabled.
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
+ withSQLConf(SQLConf.DECORRELATE_PREDICATE_SUBQUERIES_IN_JOIN_CONDITION.key ->
+ decorrelateInJoinCondition) {
+ checkAnswerAndPlan(query, Row(1) :: Nil)
+ }
+ }
+ }
+ }
+
+ test("SPARK-59351: a carried subquery is classified as on the subquery-plan route") {
+ // The carried router applies the same rules as rewriteExistentialExprInSubqueryPlan, against
+ // the inputs of the existence join rather than the outer and the subquery plan.
+ withTempView("t1", "t2", "t4", "t5", "t3n", "t5n") {
+ Seq(1, 2, 3).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq(1, 8, 9).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((9, 7)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+ Seq(Some(3), None).toDF("x").persist().createOrReplaceTempView("t3n")
+ Seq((Some(3), 7)).toDF("x", "k").persist().createOrReplaceTempView("t5n")
+
+ // A carried NOT IN is collected as the Not over the IN, so the rewrite takes its null-aware
+ // arm. Taking the positive-IN arm instead leaves NOT exists over a condition that ignores
+ // NULL comparisons: 5 NOT IN (NULL) is unknown, so the t4 row is excluded and the answer is
Review Comment:
**Nit (P3):** This case runs only over n1, n2, n4 and n5, defined just below, while t1 and t4 in the enclosing test hold different data. The excluded row is n4's (5, 100) and the 'every row' answer is n1's 1 and 2; naming those lets a reader check the explanation against the data.
**Verification:**
- **Inspection:** Every view and row named in the comment exists in the case's own fixtures and gives the stated empty versus every-row answers.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
Review Comment:
**Nit (P3):** `PlanSubqueries` only plans `ScalarSubquery` and `InSubquery(_, ListQuery(...))` (sql/core `execution/subquery.scala`), so 'plans it as an in-subquery filter and drops its own hoisted condition' describes a carried IN or NOT IN only. A carried correlated EXISTS left in this condition would reach execution as an unevaluable `Exists` and fail rather than return extra rows. Routing it is still right; the sentence should say which failure applies to which kind.
**Verification:**
- **Inspection:** The failure modes described for carried IN and for carried EXISTS match how physical sub-query planning treats each expression type.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothInputs.toSeq)
+ }
+ val unknownResult = carried.filter(sq => referencesOneInput(sq) && observesUnknown(sq))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult.toSeq)
+ }
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ carried.filter(sq => referencesOneInput(sq) && isInput(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) = rewriteExistentialExprWithAttrs(Seq(sq), p)
+ newCondition = newCondition.transform {
+ case e if e.fastEquals(sq) => rewritten.get
+ }
+ newInput
+ }
+ }
+ val newLeft = route(join.left, referencesInput(_, join.left))
+ val newRight = route(join.right, referencesInput(_, join.right))
+ if (newLeft.fastEquals(join.left) && newRight.fastEquals(join.right)) {
+ join
+ } else {
+ // Drop the `exists` attributes the nested existence joins introduced, keeping this join's
+ // own output, which already carries its own.
+ Project(join.output, join.copy(
+ left = newLeft, right = newRight, condition = Some(newCondition)))
+ }
+ }
+
+ /**
+ * Collects the existential sub-queries carried by the condition of an existence join, as the
+ * units [[rewriteExistentialExprWithAttrs]] rewrites: a NOT IN is collected as the [[Not]] over
+ * the [[InSubquery]], so the rewrite takes its null-aware arm and the unknown-result collector,
+ * which reports the same node, can be compared against it.
+ *
+ * A collected sub-query is not descended into. One nested below it is reached when it is
+ * rewritten, since the join built for it goes through the routing in turn, and one below a
+ * sub-query left in place is left with it, as everywhere else in this rule.
+ */
+ private def collectCarriedExistentialExprs(
+ expr: Expression,
+ collected: ArrayBuffer[Expression]): Unit = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return
+ }
+ expr match {
+ case sq @ Not(_: InSubquery) => collected += sq
+ case sq: Exists => collected += sq
+ case sq: InSubquery => collected += sq
+ case other => other.children.foreach(collectCarriedExistentialExprs(_, collected))
+ }
+ }
+
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan = rewriteCarriedExistentialExprs(
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint))
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint))
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = rewriteCarriedExistentialExprs(
+ Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint))
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. A nested sub-query resolves in the scope
+ * of the sub-query it is nested in, so an attribute both plans produce is one of that
+ * sub-query's own.
+ *
+ * The probes are exclusive, so swapping their order would attribute such an attribute to
+ * `outerPlan` alone: `subPlanOnly` and `bothPlans` would both be false, and a nested sub-query
+ * whose only reference is the shared attribute would be declined and left in the join condition
+ * rather than routed to the sub-query plan that can evaluate it.
+ *
+ * No plan the analyzer produces from SQL reaches this: `DeduplicateRelations` has given the
+ * sub-query its own ExprIds, and the EXISTS arms classify against the raw sub-query plan while
+ * the IN arms classify against the plan `dedupSubqueryOnSelfJoin` has already separated from the
+ * outer plan. What does reach it is the shared-attribute fixture in `RewriteSubquerySuite`,
+ * which builds the overlap directly because the analyzer is not involved there. That fixture
+ * pins the shape, not the outcome of this ordering: under either order it ends in the same
+ * error, since `dedupSubqueryOnSelfJoin`'s pre-existing conflict check reports the duplicated
+ * attribute that the hoisted condition mentions.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite here.
+ * That does not leave the sub-query unexamined: one carried by a nested sub-query that is
+ * rewritten is routed against the inputs of the join built for it, see
+ * [[rewriteCarriedExistentialExprs]].
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
Review Comment:
**Nit (P3):** This reads as covering every nested IN this method handles, but only the sub-query-plan step applies the guard: `rewriteExistentialExprInSubqueryPlan` keeps only the unknown-sensitive INs the sub-query plan owns, and the outer step below (`e => !effectivelyReferencesPlan(e, subPlan)`) still rewrites an outer-only IN two-valued, e.g. `... OR ((a IN (SELECT col1 FROM t3n)) <=> false)`. That outer behavior is the declared SPARK-59862 scope, so the sentence should be limited to sub-queries the sub-query plan owns and point at SPARK-59862.
**Verification:**
- **Inspection:** Every route the scaladoc describes as rejecting an unknown-capable nested IN applies the guard in code, and the outer-plan route is described as still two-valued.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -184,8 +188,20 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Use EXISTS if performance matters to you.
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ // The hoisted correlated predicates may carry existential sub-queries that only the
+ // sub-query plan can evaluate, see rewriteExistentialExprInJoinCondition.
+ // This arm returns a bare Join, which this rule does not offer to its own handling of
Review Comment:
Thanks. The SubquerySuite test comment now says the exemption only holds while the NOT IN is the last sub-query conjunct, but the production comment in the NOT IN arm (optimizer/subquery.scala:195-201) still says keeping the residual is 'both correct' and that a wrapped anti join is hypothetical ('were it ever wrapped'). Could you state the same condition there?
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4346076714","thread_id":"inline:4346076714","verdict_sha256":"13b761b833eea68b81b1a432a43d105b57e032a93d09a2fe8b2af5e80ef3d168"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4274278928
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala:
##########
@@ -161,4 +163,69 @@ class RewriteSubquerySuite extends PlanTest {
val existsExpected = LocalRelation(existsQuery.analyze.output).analyze
comparePlans(existsOptimized, existsExpected)
}
+
+ test("SPARK-59351: nested subquery of an EXISTS whose plan shares attributes with the outer " +
+ "plan") {
+ // The EXISTS arms route the nested subquery against the raw subquery plan, while the IN arms
+ // route it against the plan dedupSubqueryOnSelfJoin has already separated from the outer
+ // plan. A self-join can leave an ExprId shared between the two plans for the optimizer
+ // (SPARK-21835), and the analyzer is not involved here, so build that state directly: `a` is
+ // the same attribute in the outer and in the subquery plan.
+ val a = $"a".int
+ val outer = LocalRelation(a, $"b".int)
+ val sub = LocalRelation(a)
+ val nested = LocalRelation($"col1".int)
+ assert(outer.outputSet.intersect(sub.outputSet).nonEmpty,
+ "the two plans must share an attribute for this test to mean anything")
+
+ // The nested subquery references `a`, which only the subquery plan is meant to give it, and
+ // is correlated to it, so it must be rewritten against the subquery plan. Classifying `a` as
+ // a reference to the outer plan as well would report it as referencing both plans.
+ val nestedInSubquery = InSubquery(
+ Seq(a),
+ ListQuery(nested.select($"col1"), joinCond = Seq(EqualTo($"col1", a))))
+ val query = Filter(
+ Exists(sub, joinCond = Seq(Or(EqualTo($"b", a), nestedInSubquery))),
+ outer)
+
+ // The shared attribute is attributed to the subquery plan, so the nested subquery is routed
+ // there rather than reported as referencing both plans. What then reports this state is the
+ // conflict check that dedupSubqueryOnSelfJoin has always run over the semi join condition,
+ // which names the duplicated attribute: an attribute shared between the two plans that the
+ // hoisted condition mentions cannot be turned into a semi join whatever the routing does.
+ val e = intercept[AnalysisException](Optimize.execute(query))
+ assert(e.getCondition == "_LEGACY_ERROR_TEMP_1212",
+ s"expected the pre-existing conflict check to report this, got ${e.getCondition}")
+ }
+
+ test("SPARK-59351: the rewrite keeps the origin and the tags of the sub-query it replaces") {
+ // The rewrite of a nested existential sub-query builds its replacement by hand rather than
+ // through TreeNode.transformDownWithPruning, so it has to reproduce what that does: apply the
+ // rule under the origin of the node being replaced and copy its tags. Without this a runtime
+ // error raised from the rewritten condition quotes the enclosing operator instead of the
+ // sub-query fragment, which no result or plan assertion would notice.
+ val outer = LocalRelation($"a".int)
+ val inner = LocalRelation($"x".int)
+ val tag = TreeNodeTag[String]("SPARK-59351")
+ val origin = Origin(line = Some(42), startPosition = Some(7))
+ val a = outer.output.head
+ val x = inner.output.head
+ // Correlated, so that the rule's unary node handler picks the projection up at all.
+ val exists = CurrentOrigin.withOrigin(origin) {
+ Exists(inner, outerAttrs = Seq(a), joinCond = Seq(EqualTo(a, x)))
+ }
+ exists.setTagValue(tag, "kept")
+
+ // An Exists outside a Filter goes through the unary node handler, which is the path that
+ // builds the replacement by hand.
+ val rewritten = RewritePredicateSubquery(Project(Seq(Alias(exists, "e")()), outer))
+ val replacements = rewritten.expressions.flatMap(_.collect {
+ case attr: AttributeReference if attr.name == "exists" => attr
+ })
+ assert(replacements.size == 1, s"expected one exists attribute in\n$rewritten")
+ assert(replacements.head.origin == origin,
Review Comment:
Confirmed, and your prediction is exact. I added a second test covering the positive IN and NOT IN arms, then broke **only** those two arms by giving their replacements the ambient origin instead of the subquery's:
| | existing Exists test | new IN test |
|---|---|---|
| both arms correct | pass | pass |
| only IN arms broken | **pass** | **fail** |
So the old test was blind to it exactly as you said.
The new test asserts on the generated condition nodes rather than the projected attribute, since that is where they end up:
* **positive IN, uncorrelated** — the condition is the synthesized `EqualTo`, whose origin must be the subquery's;
* **NOT IN, correlated** — the condition is `And(Or(cmp, IsNull(cmp)), hoisted)`, and the combined condition, the null-aware wrapper, the comparison and the null check must all carry the subquery's origin.
It also pins the other direction, which nothing asserted before: the hoisted condition is built under a *different* origin and must keep it, since the rewrite reuses that node rather than building it, and clobbering it would misreport a user-written predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4356979479
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,625 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire and the assertion below
+ // pins it. With EXISTS the rewrite returns a Project over the join, which that handling
Review Comment:
Right, and my previous attempt at this comment still stated the handler outcome as current behaviour. I verified what actually happens before rewriting it this time: the EXISTS form raises `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY` under both values of the configuration, from the same rejection in `rewriteExistentialExprInSubqueryPlan`. The comment now says that, and puts the handler outcome as a counterfactual
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4380513465
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
Review Comment:
Fixed using the shape you described: a third-level `c1 IN (SELECT x FROM t5 WHERE x = k)` inside a nested `c1 IN (SELECT col1 FROM t4 WHERE k = c1 OR ...)`. The compared value comes from the left input of the existence join built for the nested IN and the hoisted `x = k` reads the right one. The test asserts `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY` under both values of `decorrelatePredicateSubqueriesInJoinPredicate.enabled`. The both-inputs rejection still applies on every route.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5853675261
@cloud-fan @dongjoon-hyun @LuciferYang all review comments are addressed, could
you take another look? Below are some of the points where i would require your inputs to take a decision. PTAL. Thanks!
1. **@cloud-fan — the preparation step for self-join deduplication (`r4068594134`).** Routing now
classifies as if the subplan had been deduplicated, which is the part I could do without
rewriting the hoisted condition. The colliding EXISTS shape still ends at the pre-existing
`_LEGACY_ERROR_TEMP_1212` from `buildJoin` rather than succeeding. Do you want the full
preparation step — deduplication plus condition rewriting, across all four arms — in this PR, or
as a follow-up?
2. **@dongjoon-hyun — phase 2 accepting on a negation (`r4065967251`).** Your fail-safe positive
test and cloud-fan's `r4057052746` pull in opposite directions: requiring a positive reason to
rewrite on the outer plan would re-reject the subquery whose retained outer attributes no longer
correspond to a condition, which `r4057052746` asked to keep rewriting there. I documented the
invariant instead. Which of the two would you rather have?
3. **@dongjoon-hyun — the reused-ExprId concern (`r4065967215` (a)).** I could not build a query
that reaches the optimizer with the conflict intact; every shape I tried, including your
`c1 IN (SELECT a FROM t1)`, comes out of the analyzer deduplicated, and the conflict check in
`buildJoin` rejects the rest first. If you have a query that gets there, I will add it as a test
and fix what it shows.
4. **@dongjoon-hyun — the asymmetry inside `dedupSubqueryOnSelfJoin`** (able to throw when
`buildJoin` passes the condition, only aliasing when the IN arms call it without one). Left as is,
being pre-existing and the same work as question 1. Address it here, or separately?
5. **Everyone — the two new error condition names and their wording.** They are my choice; happy to
rename or merge them into one sub-condition if you prefer.
6. **@cloud-fan — the follow-up you asked me to track.** Replacing a three-valued `IN` with a
non-nullable existence bit is already wrong on released 4.0.1 on the outer and projection routes,
with no correlation involved:
`SELECT a, ((a IN (SELECT col1 FROM t3)) <=> false) FROM t1` returns `true, true, false` where
PostgreSQL and DuckDB return `false, false, false`. Shall I file that as a separate JIRA with
these repros and the reconstruction sketch — two existence joins, the second on
`cond OR isnull(cond)` so multi-column row comparison stays correct, plus a copy of the subquery
plan with fresh expression ids?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114367048
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3227,284 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // Checks the result, and that every node of the optimized plan can produce the attributes
+ // it references. The latter is what this fix is about: an existence join whose condition
+ // references an attribute produced by neither of its children can still return the right
+ // answer when the nested relation is empty at runtime, because the invalid condition is
+ // then never bound. checkAnswer alone would not catch it, as the missing input checks it
+ // runs only look at the root of the plan.
+ def checkAnswerAndPlan(query: String, expected: Seq[Row]): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ checkAnswer(df, expected)
+ }
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // would drop its correlated predicate col1 = c1 and return an extra row, so it is rejected.
Review Comment:
You are right, and the claim was stale: I verified it on the EXISTS form, then changed the outer
predicate to NOT IN for LuciferYang's discrimination point without re-checking the data. The
comment no longer claims a result change for these rows; it states that the correlated predicate is
dropped, and notes that the EXISTS form is what demonstrates the extra row. The assertion, which is
on the rejection, is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4114399351
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,171 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans can be rewritten into an existence join on
+ * neither side, so it is left in the join condition, where both plans are in scope. Leaving it
+ * there is only correct while it is uncorrelated: the join condition of a correlated one is
+ * dropped when it is planned as an in-subquery filter, which would silently change the result,
+ * so a correlated one is reported as unsupported instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration; it
+ * survives only with `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate`
+ * disabled.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
Review Comment:
Documented rather than changed, and I would value your view. A positive test is not available
without also refusing the sub-queries whose retained outer attributes no longer correspond to a
condition — the shape cloud-fan asked to keep routing to the outer plan in `r4057052746` — so the
two asks pull in opposite directions. The invariant the routing rests on is now named at the call
site: `effectiveReferences` must see every attribute the rewrite can put in a join condition. If you
would rather have the fail-safe test and accept re-rejecting that shape, say so and I will switch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4278473903
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -158,20 +159,23 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Filter the plan by applying left semi and left anti joins.
withSubquery.foldLeft(newFilter) {
case (p, Exists(sub, _, _, conditions, subHint)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftSemi, joinCond, subHint)
Project(p.output, join)
case (p, Not(Exists(sub, _, _, conditions, subHint))) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, sub, joinCond),
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(conditions, p, sub)
+ val join = buildJoin(outerPlan, rewriteDomainJoinsIfPresent(outerPlan, newSub, joinCond),
LeftAnti, joinCond, subHint)
Project(p.output, join)
case (p, InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val (joinCond, outerPlan) = rewriteExistentialExpr(inConditions ++ conditions, p)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ val (joinCond, outerPlan, newSub) =
+ rewriteExistentialExprInJoinCondition(inConditions ++ conditions, p, dedupSub)
Review Comment:
Added as a seperate commit so the routing fix and the estimation change stay separately reviewable.
### Both statistics modes
`SizeInBytesOnlyStatsPlanVisitor.visitJoin` gains an `ExistenceJoin` arm that rescales the left child's size the same way `visitUnaryNode` does, since the row count is unchanged and only the appended Boolean widens each row:
```scala
case ExistenceJoin(_) =>
visitExistenceJoin(p)
```
`JoinEstimation` gains `estimateExistenceJoin()`, which takes the row count from the left child and sizes the output from `join.output` — that is the left output plus `exists`, so the appended width is accounted for, and the attribute has no column statistics so `getSizePerRow` falls back to its type's default size. Only the left child's attribute statistics are propagated; the right child's describe attributes an existence join does not output.
It requires only `rowCountsExist(join.left)`, so a right child with no statistics, or an empty one, is still estimated rather than falling back to the product.
### Measured
On the shape from this thread, with `spark.sql.cbo.enabled` either way:
| | left | right | node estimate |
|---|---|---|---|
| before | 2,579 | 2,591 | **6,682,189** |
| after | 2,579 | 2,591 | **2,601** |
The old estimate is the product exactly — a 2,591x over-estimate, the right child's size used as a multiplier — for an operator that emits precisely the left child's 2,579 bytes plus one byte per row.
### A TPC-DS plan actually improves
`q10.sf100` under `TPCDSV1_4_PlanStabilityWithStatsSuite` moves, and it is the regression you predicted, in an approved plan. Net operator change:
```
-1 SortMergeJoin +1 BroadcastHashJoin
-2 Exchange +1 BroadcastExchange
-2 Sort
-3 WholeStageCodegen
```
The subtree carrying `Filter [exists,exists]` is now small enough to broadcast, so the shuffle and sort of the customer-side input disappear:
```
- SortMergeJoin [c_current_cdemo_sk,cd_demo_sk]
- ... Sort [c_current_cdemo_sk]
- ... Exchange [c_current_cdemo_sk] #2
+ BroadcastHashJoin [c_current_cdemo_sk,cd_demo_sk]
+ ... BroadcastExchange #2
```
Golden files regenerated with `SPARK_GENERATE_GOLDEN_FILES=1`, not edited. This is the only one of the 8 approved plans containing an `ExistenceJoin` that changes, and only in the CBO-enabled suite — which fits, since that is the mode where `JoinEstimation` used to return `None`.
### Regression
The new plan-selection test in `SubquerySuite` is the one you asked for: a nested inner-routed existence join whose left child is below the broadcast threshold while the product would exceed it, asserting the enclosing equi semi join keeps its broadcast. It is marked `DisableAdaptiveExecution`, since with AQE on runtime statistics recover the plan and the estimate under test is not the one the planner uses. It also notes why the hoisted predicate is `a > c1` rather than the key equality: with a disjunction as the whole condition the join is a broadcast nested loop join whatever the statistics say, so it could not show the effect.
Checked that it discriminates — with the two new arms reverted it fails with
```
6682189 was not less than 5158 the existence join is estimated at 6682189 from a
left child of 2579 and a right child of 2591
```
Statistics regressions cover both modes: `existence join` and `existence join with an empty or unestimated right child` in `JoinEstimationSuite` pin the CBO row count, size and left-only attribute stats, including a right child with a zero row count and one with no column statistics; `existence join is estimated from its left child in both statistics modes` in `BasicStatsEstimationSuite` pins the same invariant through `checkStats` with CBO on and off, and asserts the estimate is not the product.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4274475049
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,412 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
Review Comment:
Addressed — sorry this thread went unanswered while the fix landed; it has been on the branch since `ab77ebe5`.
Confirmed, and the composition is worse than silently wrong rows in the example I could build — it crashes. The gap is exactly where you place it: the three passes do not cover each other.
* the both-plans rejection only matched a sub-query that is *itself* correlated and references both plans;
* the unknown-result rejection only matches a nullable IN whose own value is observed beyond truth;
* the rewrite declines the both-plan parent and, since the fix for `r4022391750`, does not descend into it.
A correlated IN owned by the sub-query plan, nested in that parent, fell through all three: its own effective references are inner-only, so it looks routable, but nothing reaches it, and `PlanSubqueries` plans the `ListQuery` as an in-subquery filter and drops the predicate `PullupCorrelatedPredicates` hoisted into it. The no-descent behaviour is right; what was missing is that the validation did not know about that boundary.
Reproduced with `t1(a) = 1,2,3`, `t2(c1) = 1,8,9`, `t3(col1) = 3,9` and `t4(col1, k) = (1,9),(9,1)`, where the correlation is decisive and no NULL is involved:
```sql
SELECT * FROM t1 WHERE EXISTS (
SELECT 1 FROM t2
WHERE a = c1
OR (a + (CASE WHEN c1 IN (SELECT col1 FROM t4 WHERE k = c1) THEN 1 ELSE 0 END))
IN (SELECT col1 FROM t3));
```
PostgreSQL and DuckDB return `1, 3`; dropping the correlated predicate returns `1, 2, 3`. Before the fix, with `decorrelatePredicateSubqueriesInJoinPredicate.enabled=false`, it failed with
```
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class
org.apache.spark.sql.catalyst.InternalRow
```
and under the default configuration the pre-existing join-condition rewrite rejected the parent first, which is why nothing caught this.
### Fix is as follows:
A sub-query that stays in the join condition stays there whole, so the both-plans rejection now also applies when such a sub-query *carries* a correlated existential descendant:
```scala
case sq @ (_: Exists | _: InSubquery)
if referencesBothPlans(sq) &&
(hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(sq)) => sq
```
`carriesCorrelatedSubquery` looks strictly below the node, since the node itself is already classified by `hasCorrelatedCondition`, and reports any `Exists` or `InSubquery` with a non-empty hoisted condition. Reporting the enclosing sub-query keeps `NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY` accurate, as that is the sub-query which references both plans and cannot be supported, and it is the one the user wrote.
Against your success criteria: no correlated descendant blocked beneath a declined parent reaches physical planning; the distinguishing example now raises the structured error under both settings of that configuration rather than crashing; and the behaviours you asked to preserve are unchanged — the uncorrelated both-plan residual with no correlated descendant still returns `1, 2`, the simplified-away correlation case is still rewritten against the outer plan, and direct routing and the nullable IN/NOT IN handling are untouched, each still covered by its own assertion.
I did not take the carrier-aware traversal from your recommended change. Rejecting the carrier rather than classifying each descendant as reachable or blocked gives the same guarantee here with much less machinery, and it cannot reject a descendant the rewrite would have consumed, since it only fires for a sub-query that is itself staying in the condition. If you would rather have the reachability state threaded through the walk, say so and I will do that instead.
One scope note, since it came up in `r4257086397`: that later finding is a *different* boundary — a rewritten subplan-owned parent rather than a declined both-plan one — and I have answered it in its own thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4286926784
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,476 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. The two plans share no ExprId in any
+ * shape reached by the suite: `DeduplicateRelations` has given the sub-query its own, and the
+ * EXISTS arms classify against the raw sub-query plan while the IN arms classify against the
+ * plan `dedupSubqueryOnSelfJoin` has already separated from the outer plan. The order is
+ * therefore defensive rather than something a query exercises, and it is the safe one: a
+ * nested sub-query resolves in the scope of the sub-query it is nested in, so an attribute
+ * both plans produce is one of that sub-query's own, and attributing it to `outerPlan` instead
+ * would let a single shared attribute satisfy both sides of the classification and have a
+ * nested sub-query referencing only the inner query rejected as referencing both.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite
+ * without rejecting the sub-query, as a sub-query referencing neither plan can be left in the
+ * join condition unharmed.
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
Review Comment:
Thanks for your inputs @cloud-fan, I fixed by routing the carried descendant against the newly built join, rather than by rejecting it or by making the validation configuration-dependent. (option 1)
`rewriteCarriedExistentialExprs` now runs on every existence join this rule builds, in all three arms. If the grafted condition carries a correlated existential subquery, it is rewritten against whichever of that join's own inputs can evaluate it; the routing recurses, since the joins it builds go through it in turn. An uncorrelated one is deliberately left alone -- it has no hoisted condition to lose and `InSubqueryExec` evaluates it three-valued, which is also what `r4281437494` is about. One that would need both inputs, or whose unknown is observable, is rejected.
Both of your sub-cases are fixed, and the two configurations now agree:
| | `decorrelate...InJoinCondition=false` | `=true` |
|---|---|---|
| master `6e5c61fb` | `INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND` | `INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND` |
| before this commit | `1, 2, 3` | `1` |
| now | `1` | `1` |
DuckDB returns `1`. The plan under the disabled configuration is now the one the enabled path already produced:
```
+- Join ExistenceJoin(exists#39), ((c1#12 = col1#23) AND ((k#24 = c1#12) OR exists#40))
:- t2
+- Join ExistenceJoin(exists#40), ((col1#23 = col1#32) AND (col1#32 = k#24))
:- t4
+- t5
```
That is the point of doing it here rather than relying on the join-condition handling: that handling is gated on the configuration, which is exactly why the two disagreed.
New test `existential subquery carried by a rewritten nested subquery` runs the shape under both settings; it fails with the routing disabled, returning `1, 2, 3`. The scaladoc on `planReferences` no longer claims a subquery referencing neither plan "can be left in the join condition unharmed", which was the assumption behind the defect; it now points at the carried-routing instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4286934055
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -184,8 +188,12 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Use EXISTS if performance matters to you.
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ // The hoisted correlated predicates may carry existential sub-queries that only the
+ // sub-query plan can evaluate, see rewriteExistentialExprInJoinCondition.
+ val (subConditions, newSub) =
+ rewriteExistentialExprInSubqueryPlan(conditions, p, dedupSub)
Review Comment:
Confirmed and fixed. I verified the regression against the true merge base this time:
| predicate (outer `NOT IN`) | master `6e5c61fb` | before this commit | now | DuckDB |
|---|---|---|---|---|
| `(c1 IN (SELECT col1 FROM t3n)) <=> false` | `2, 3` | `0A000` | `2, 3` | `2, 3` |
| `(c1 IN (...)) IS NULL` | `2, 3` | `0A000` | `2, 3` | `2, 3` |
| `COALESCE(c1 IN (...), true)` | `2, 3` | `0A000` | `2, 3` | `2, 3` |
| `(c1 NOT IN (...)) <=> false` | `2, 3` | `0A000` | `2, 3` | `2, 3` |
`rewriteExistentialExprInSubqueryPlan` takes a `keepsThreeValuedResiduals` flag, passed only by the NOT IN arm. Under it an uncorrelated, unknown-sensitive, subquery-plan-owned nested IN is partitioned out of the rejection and excluded from `canRewrite`, so it stays in the anti-join condition exactly as at the merge base. Correlated ones -- including ones that only carry a correlated descendant -- are still rejected, and truth-only ones are still rewritten.
As you asked, the exemption is tied to the bare-`Join` result in a comment at the call site, so a future change that wraps it and brings the join-condition handling into reach does not silently start two-valuing these residuals.
New test `NOT IN keeps an uncorrelated three-valued residual` covers all four shapes, plus a correlated nested IN under both an outer `NOT IN` and an outer `EXISTS` still being rejected, and a truth-only nested IN under `NOT IN` still being rewritten. With the exemption removed it fails on the first shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4286956777
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +406,481 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Same as [[rewriteExistentialExpr]], but it also returns the newly introduced attributes, and
+ * it only rewrites the existential sub-queries for which `canRewrite` returns true. A sub-query
+ * that is not rewritten stays in the returned expression as it is, and is not descended into:
+ * rewriting an existential sub-query nested in its join condition would graft an existence join
+ * onto the plan for an `exists` reference that the sub-query left in place may never evaluate.
+ */
private def rewriteExistentialExprWithAttrs(
exprs: Seq[Expression],
- plan: LogicalPlan): (Option[Expression], LogicalPlan, Seq[Attribute]) = {
+ plan: LogicalPlan,
+ canRewrite: Expression => Boolean = _ => true): (Option[Expression], LogicalPlan,
+ Seq[Attribute]) = {
var newPlan = plan
val introducedAttrs = ArrayBuffer.empty[Attribute]
- val newExprs = exprs.map { e =>
- e.transformDownWithPruning(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
- case Exists(sub, _, _, conditions, subHint) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- val existenceJoin = ExistenceJoin(exists)
- val newCondition = conditions.reduceLeftOption(And)
- newPlan =
- buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
- existenceJoin, newCondition, subHint)
- introducedAttrs += exists
- exists
- case Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- // To handle a null-aware predicate not-in-subquery in nested conditions
- // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
- // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
- //
- // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
- // is transformed into a plan below;
- // == Optimized Logical Plan ==
- // Project [id#78, v#79]
- // +- Filter ((v#79 > 0) OR NOT exists#83)
- // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
- // :- Relation[id#78,v#79] parquet
- // +- Relation[id#80] parquet
- val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
- val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan,
- rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
- ExistenceJoin(exists), Some(finalJoinCond), joinHint)
- introducedAttrs += exists
- Not(exists)
- case InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)) =>
- val exists = AttributeReference("exists", BooleanType, nullable = false)()
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
- val joinHint = JoinHint(None, subHint)
- newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
- ExistenceJoin(exists), newConditions, joinHint)
- introducedAttrs += exists
- exists
+ // Builds the replacement of a sub-query expression under that expression's origin and with
+ // its tags, as TreeNode.transformDownWithPruning does for the nodes a rule replaces. Without
+ // it every node built below, the `exists` attribute and the join conditions included, would
+ // take the origin of the enclosing operator, and a runtime error raised from the rewritten
+ // condition would quote that operator rather than the sub-query fragment.
+ def replacing(sq: Expression)(replacement: => Expression): Expression = {
+ val newExpr = CurrentOrigin.withOrigin(sq.origin)(replacement)
+ newExpr.copyTagsFrom(sq)
+ newExpr
+ }
+ def rewrite(expr: Expression): Expression = {
+ if (!expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY)) {
+ return expr
+ }
+ expr match {
+ case sq @ Exists(sub, _, _, conditions, subHint) if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ val existenceJoin = ExistenceJoin(exists)
+ val newCondition = conditions.reduceLeftOption(And)
+ newPlan =
+ buildJoin(newPlan, rewriteDomainJoinsIfPresent(newPlan, sub, newCondition),
+ existenceJoin, newCondition, subHint)
+ introducedAttrs += exists
+ exists
+ }
+ case sq @ Not(InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint)))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ // To handle a null-aware predicate not-in-subquery in nested conditions
+ // (e.g., `v > 0 OR t1.id NOT IN (SELECT id FROM t2)`), we transform
+ // `inCondition` (t1.id=t2.id) into `(inCondition) OR ISNULL(inCondition)`.
+ //
+ // For example, `SELECT * FROM t1 WHERE v > 0 OR t1.id NOT IN (SELECT id FROM t2)`
+ // is transformed into a plan below;
+ // == Optimized Logical Plan ==
+ // Project [id#78, v#79]
+ // +- Filter ((v#79 > 0) OR NOT exists#83)
+ // +- Join ExistenceJoin(exists#83), ((id#78 = id#80) OR isnull((id#78 = id#80)))
+ // :- Relation[id#78,v#79] parquet
+ // +- Relation[id#80] parquet
+ val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
+ val finalJoinCond = (nullAwareJoinConds ++ conditions).reduceLeft(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan,
+ rewriteDomainJoinsIfPresent(newPlan, newSub, Some(finalJoinCond)),
+ ExistenceJoin(exists), Some(finalJoinCond), joinHint)
+ introducedAttrs += exists
+ Not(exists)
+ }
+ case sq @ InSubquery(values, ListQuery(sub, _, _, _, conditions, subHint))
+ if canRewrite(sq) =>
+ replacing(sq) {
+ val exists = AttributeReference("exists", BooleanType, nullable = false)()
+ // Deduplicate conflicting attributes if any.
+ val newSub = dedupSubqueryOnSelfJoin(newPlan, sub, Some(values))
+ val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val newConditions = (inConditions ++ conditions).reduceLeftOption(And)
+ val joinHint = JoinHint(None, subHint)
+ newPlan = Join(newPlan, rewriteDomainJoinsIfPresent(newPlan, newSub, newConditions),
+ ExistenceJoin(exists), newConditions, joinHint)
+ introducedAttrs += exists
+ exists
+ }
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ // Iterates the references and probes the output set, rather than materialising the
+ // intersection: AttributeSet.intersect builds a set from its argument, so the operand order
+ // there would decide whether this walks the references or the whole output of `plan`. The
+ // membership probes still construct wrappers, so this is cheaper, not allocation-free.
+ e.references.exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Returns true if `e` effectively references any of the attributes produced by `plan`, see
+ * [[effectiveReferences]].
+ */
+ private def effectivelyReferencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ effectiveReferences(e).exists(plan.outputSet.contains)
+ }
+
+ /**
+ * Which of an outer plan and a sub-query plan a set of references can be evaluated by, as
+ * [[planReferences]] determines it.
+ */
+ private case class PlanReferences(subPlanOnly: Boolean, bothPlans: Boolean)
+
+ /**
+ * Classifies `references` against `outerPlan` and `subPlan` in a single pass: whether they can
+ * only be evaluated by `subPlan`, and whether they need both plans.
+ *
+ * An attribute that both plans produce is attributed to `subPlan` rather than to `outerPlan`,
+ * which is what the `subPlan` probe coming first does. A nested sub-query resolves in the scope
+ * of the sub-query it is nested in, so an attribute both plans produce is one of that
+ * sub-query's own; attributing it to `outerPlan` instead would let a single shared attribute
+ * satisfy both sides of the classification and have a nested sub-query referencing only the
+ * inner query rejected as referencing both.
+ *
+ * No plan the analyzer produces from SQL reaches this: `DeduplicateRelations` has given the
+ * sub-query its own ExprIds, and the EXISTS arms classify against the raw sub-query plan while
+ * the IN arms classify against the plan `dedupSubqueryOnSelfJoin` has already separated from the
+ * outer plan. What does reach it is the shared-attribute fixture in `RewriteSubquerySuite`,
+ * which builds the overlap directly because the analyzer is not involved there. That fixture
+ * pins the shape, not the outcome of this ordering: `dedupSubqueryOnSelfJoin`'s pre-existing
+ * conflict check reports the duplicated attribute first, so it ends in the same error whichever
+ * plan the shared attribute is attributed to.
+ *
+ * An attribute neither plan produces leaves both answers false, which declines the rewrite
+ * without rejecting the sub-query, as a sub-query referencing neither plan can be left in the
+ * join condition unharmed.
+ */
+ private def planReferences(
+ references: AttributeSet,
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): PlanReferences = {
+ var subPlanReference = false
+ var outerPlanOnlyReference = false
+ val attrs = references.iterator
+ // Stops as soon as both plans are known to be referenced, which is all the two answers need.
+ while (attrs.hasNext && !(subPlanReference && outerPlanOnlyReference)) {
+ val attr = attrs.next()
+ if (subPlan.outputSet.contains(attr)) {
+ subPlanReference = true
+ } else if (outerPlan.outputSet.contains(attr)) {
+ outerPlanOnlyReference = true
+ }
+ }
+ PlanReferences(
+ subPlanOnly = subPlanReference && !outerPlanOnlyReference,
+ bothPlans = subPlanReference && outerPlanOnlyReference)
+ }
+
+ /**
+ * The attributes that `e` references and can still evaluate. For an existential sub-query
+ * expression these are the ones of the values it compares and of the correlated condition that
+ * [[PullupCorrelatedPredicates]] hoisted into it, rather than the ones of `references`, which
+ * also include the outer attributes the pull-up retains for idempotency. Those outlive the
+ * condition that referenced them once BooleanSimplification has eliminated it, as in
+ * `a IN (SELECT col1 FROM t3 WHERE false AND col1 = c1)`, which keeps `c1` as an outer
+ * attribute of a sub-query that no longer reads it, and which must still be rewritten against
+ * the outer plan rather than treated as referencing the sub-query plan.
+ *
+ * The sets are merged with `AttributeSet.fromAttributeSets` rather than built from a flattened
+ * sequence of attributes: `AttributeSet.apply` would take one allocation per attribute of a
+ * sequence as long as the total number of references, which a row-valued IN over a wide row
+ * makes unnecessarily large.
+ */
+ private def effectiveReferences(e: Expression): AttributeSet = e match {
+ case Exists(_, _, _, joinCond, _) =>
+ AttributeSet.fromAttributeSets(joinCond.map(_.references))
+ case Not(in: InSubquery) => effectiveReferences(in)
+ case InSubquery(values, ListQuery(_, _, _, _, joinCond, _)) =>
+ AttributeSet.fromAttributeSets(values.map(_.references) ++ joinCond.map(_.references))
+ case _ => e.references
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, since neither alone can evaluate it, so it has to stay in the join condition,
+ * where both plans are in scope. Leaving it there is only correct while it is uncorrelated: the
+ * join condition of a correlated one is dropped when it is planned as an in-subquery filter,
+ * which would silently change the result, so a correlated one is reported as unsupported
+ * instead. Note that such a sub-query can be
+ * correlated only to the sub-query plan, as being correlated to the outer plan as well would
+ * require two levels of correlation, which the Analyzer rejects.
+ *
+ * A sub-query left here is further subject to the rewrite of predicate sub-queries in join
+ * conditions, which rejects one referencing both plans under the default configuration. It
+ * survives that rewrite when
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled` is disabled, and
+ * also when it is an uncorrelated IN sub-query and
+ * `spark.sql.optimizer.optimizeUncorrelatedInSubqueriesInJoinCondition.enabled` is disabled,
+ * since it is then not among the sub-queries that rewrite considers.
+ *
+ * An existence join yields only whether a row matched, so its `exists` attribute cannot tell
+ * FALSE from unknown, while `IN` is three-valued. The two are indistinguishable while the value
+ * only feeds a predicate, which is what a hoisted condition normally does, but not when it
+ * reaches something else, e.g. `(c1 IN (SELECT col1 FROM t3)) <=> false`, which is FALSE for a
+ * NULL that matches nothing and TRUE for the `exists` attribute. An IN sub-query whose row
+ * comparison can evaluate to unknown is therefore rejected in that position rather than
+ * rewritten. NOT IN is rewritten with a null-aware join condition of its own, which is equally
+ * two-valued, so it is rejected there too.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any. This accepts on the
+ // negation of the test above rather than on a positive one, which rests on the invariant that
+ // `effectiveReferences` sees every attribute the rewrite can put in a join condition: one it
+ // cannot see would be rewritten here and end up in a condition the outer plan cannot evaluate.
+ // A positive test is not available without also refusing the sub-queries whose retained outer
+ // attributes no longer correspond to a condition, which must keep being rewritten here.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !effectivelyReferencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // The rejection collector below is a plain `collect`, which visits every node of the hoisted
+ // conditions whether or not it can hold a sub-query, so skip the classification entirely when
+ // they hold none at all, which is the common case for a correlated EXISTS or IN.
+ // [[collectUnknownSensitiveInSubqueries]] prunes subquery-free subtrees on its own, so this
+ // guard only saves it the top-level call.
+ if (!conditions.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return (conditions.reduceOption(And), subPlan)
+ }
+ // A sub-query left in the join condition loses its own join condition when it is planned
+ // there, so a correlated one would silently return a wrong result: reject it instead. The
+ // same holds for a correlated sub-query nested below one that is left there, since the
+ // rewrite leaves a declined sub-query whole, children included, and nothing else will reach
+ // it: see carriesCorrelatedSubquery. Note that this walks the whole expression, including the
+ // join condition of a sub-query the rewrite declines to descend into, which is deliberate for
+ // the same reason.
+ //
+ // Which plan a candidate references is asked of it up to three times below, once for the
+ // rejection of a sub-query referencing both plans, once to keep only the unknown-sensitive
+ // ones the sub-query plan owns, and once more by the rewrite itself. Both answers come from
+ // the same reference set in a single pass over it, and the result is memoised per candidate,
+ // so neither the set nor the scan is repeated. The memo is keyed by identity: the candidates
+ // are nodes of the expressions being classified, and comparing two of them structurally would
+ // compare the sub-query plans they hold.
+ val classified = new java.util.IdentityHashMap[Expression, PlanReferences]()
+ def classify(sq: Expression): PlanReferences = {
+ var references = classified.get(sq)
+ if (references == null) {
+ references = planReferences(effectiveReferences(sq), outerPlan, subPlan)
+ classified.put(sq, references)
+ }
+ references
+ }
+ def ownedBySubPlan(sq: Expression): Boolean = classify(sq).subPlanOnly
+ def referencesBothPlans(sq: Expression): Boolean = classify(sq).bothPlans
+ val referencingBothPlans = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if referencesBothPlans(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(sq)) => sq
+ })
+ if (referencingBothPlans.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothPlans)
+ }
+ // An IN sub-query whose result can be unknown cannot be represented by the `exists` attribute
+ // of an existence join once that result is observable, see above.
+ val unknownResult = conditions
+ .flatMap(unknownSensitiveInSubqueries(_, truthOnly = true))
+ .filter(ownedBySubPlan)
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult)
+ }
+ val (newCond, newSubPlan, _) =
+ rewriteExistentialExprWithAttrs(conditions, subPlan, ownedBySubPlan)
+ (newCond, newSubPlan)
+ }
+
+ /**
+ * Collects the IN sub-queries in `expr` whose result can be unknown and whose value is observed
+ * beyond being TRUE, so that rewriting them into an existence join, whose `exists` attribute is
+ * FALSE where IN is unknown, would be observable. See [[rewriteExistentialExprInJoinCondition]].
+ *
+ * `truthOnly` states whether the value of `expr` is only ever tested for being TRUE, which is
+ * where unknown and FALSE cannot be told apart. That holds for the operands of AND and OR within
+ * a condition that ends up in a Filter or a join condition, and for the positions
+ * [[ReplaceNullWithFalseInPredicate]] rewrites for the same reason: the operand of an
+ * `EqualNullSafe` against TRUE, and the predicate of an `If` or of a `CaseWhen` branch. The
+ * result branches of a conditional are consumed wherever the conditional itself is, so they
+ * inherit its context rather than being value positions of their own.
+ *
+ * It does not hold under NOT, where `NOT FALSE` is TRUE while `NOT unknown` is unknown. A NOT
+ * directly over an IN sub-query is still safe, but for another reason: it is rewritten with a
+ * null-aware join condition of its own, which is TRUE where the comparison is unknown, so the
+ * negation of its `exists` attribute is FALSE there, as SQL requires.
+ */
+ private def unknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean): Seq[Expression] = {
+ val reported = ArrayBuffer.empty[Expression]
+ collectUnknownSensitiveInSubqueries(expr, truthOnly, reported)
+ reported.toSeq
+ }
+
+ /**
+ * Appends what [[unknownSensitiveInSubqueries]] reports to `reported`, in the order the
+ * expression is walked. The results are accumulated rather than concatenated on the way back up,
+ * so a condition with many reportable sub-queries does not copy the prefix at every node.
+ */
+ private def collectUnknownSensitiveInSubqueries(
+ expr: Expression,
+ truthOnly: Boolean,
+ reported: ArrayBuffer[Expression]): Unit = expr match {
+ case _ if !expr.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY) =>
+ case And(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ case Or(left, right) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(right, truthOnly, reported)
+ // `null <=> true` and `false <=> true` are both FALSE, so only truth is observed here whatever
+ // consumes the comparison.
+ case EqualNullSafe(left, Literal.TrueLiteral) =>
+ collectUnknownSensitiveInSubqueries(left, truthOnly = true, reported)
+ case EqualNullSafe(Literal.TrueLiteral, right) =>
+ collectUnknownSensitiveInSubqueries(right, truthOnly = true, reported)
+ // COALESCE(x, false) maps both unknown and FALSE to FALSE, so only truth is observed. A
+ // different default does not: COALESCE(x, true) is TRUE where x is unknown and FALSE where x
+ // is FALSE.
+ case Coalesce(child +: rest) if rest.headOption.contains(Literal.FalseLiteral) =>
+ collectUnknownSensitiveInSubqueries(child, truthOnly = true, reported)
+ case If(predicate, trueValue, falseValue) =>
+ collectUnknownSensitiveInSubqueries(predicate, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(trueValue, truthOnly, reported)
+ collectUnknownSensitiveInSubqueries(falseValue, truthOnly, reported)
+ case CaseWhen(branches, elseValue) =>
+ branches.foreach { case (condition, value) =>
+ collectUnknownSensitiveInSubqueries(condition, truthOnly = true, reported)
+ collectUnknownSensitiveInSubqueries(value, truthOnly, reported)
+ }
+ elseValue.foreach(collectUnknownSensitiveInSubqueries(_, truthOnly, reported))
+ // A NOT directly over an IN sub-query keeps its own null-aware rewrite, so only its operands
Review Comment:
Correct. The arm calls `collectOrDescend(not, in, truthOnly, reported)`, which reports the node in a value position and only descends otherwise, so "only its operands are examined" was wrong -- and the test asserting a reported expression beginning with `(NOT ` depends on the reporting. Have rewritten
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6199967506
Thanks for the additional comments @cloud-fan. I have addressed them, as well as updated the PR description to point out the latest additions as part of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4040714428
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -2678,4 +2678,133 @@ class SubquerySuite extends SharedSparkSession
assert(exposedAttribute.exprId == outerReferenceAttribute.exprId)
}
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n") {
+ Seq((1), (2), (3), (7)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ val df1 = sql(query1)
+ // Every plan node must be able to produce the attributes it references.
+ val invalidNodes = df1.queryExecution.optimizedPlan.collect {
+ case p if p.missingInput.nonEmpty => p
+ }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |${df1.queryExecution.optimizedPlan}""".stripMargin)
+ checkAnswer(df1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // Same, with a nested subquery that returns no matching row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswer(sql(query2), Row(1) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswer(sql(query3), Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswer(sql(query4), Row(1) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR EXISTS (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswer(sql(query5), Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is
+ // never true, so only the correlated predicate can be satisfied.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswer(sql(query6), Row(1) :: Nil)
+
+ // Without the NULL, c1 NOT IN (3, 9) holds for c1 = 1.
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswer(sql(query7), Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested subquery that is itself correlated to the query it is nested in.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswer(sql(query8), Row(1) :: Nil)
Review Comment:
Agreed, and your reasoning about why the built-in checks do not cover it,
have updated such that `checkAnswerAndPlan(query, expected)` helper used by all eight queries
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5720404819
Thanks @LuciferYang for the comments. Have updated the PR to address all four comments. PTAL Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4047869684
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,100 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, so it is left in the join condition, where both plans are in scope. This is only
+ * correct for an uncorrelated sub-query: the join condition of a correlated one is dropped when
+ * it is planned as an in-subquery filter, which would silently change the result, so it is
+ * reported as unsupported instead. Note that such a sub-query can be correlated only to the
+ * sub-query plan, as being correlated to the outer plan as well would require two levels of
+ * correlation, which the Analyzer rejects.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan, LogicalPlan) = {
+ val (subCond, newSubPlan) =
+ rewriteExistentialExprInSubqueryPlan(conditions, outerPlan, subPlan)
+ // The sub-queries that do not reference the sub-query plan are rewritten against the outer
+ // plan, as they only reference attributes of the outer plan, if any.
+ val (newCond, newOuterPlan, _) = rewriteExistentialExprWithAttrs(
+ subCond.toSeq, outerPlan, e => !referencesPlan(e, subPlan))
+ (newCond, newOuterPlan, newSubPlan)
+ }
+
+ /**
+ * Rewrites the existential sub-queries in `conditions` that can only be evaluated by the
+ * sub-query plan, that is those referencing the sub-query plan but not the outer plan, into
+ * existence joins on top of the sub-query plan. See
+ * [[rewriteExistentialExprInJoinCondition]] for details.
+ *
+ * Returns the rewritten condition along with the updated sub-query plan.
+ */
+ private def rewriteExistentialExprInSubqueryPlan(
+ conditions: Seq[Expression],
+ outerPlan: LogicalPlan,
+ subPlan: LogicalPlan): (Option[Expression], LogicalPlan) = {
+ // A correlated sub-query referencing both plans has to stay in the join condition, where its
+ // own join condition is lost, so reject it rather than silently return a wrong result.
+ val unsupported = conditions.flatMap(_.collect {
+ case sq @ (_: Exists | _: InSubquery)
+ if isCorrelatedSubquery(sq) &&
+ referencesPlan(sq, subPlan) && referencesPlan(sq, outerPlan) => sq
+ })
+ if (unsupported.nonEmpty) {
+ throw QueryCompilationErrors.unsupportedCorrelatedSubqueryInJoinConditionError(unsupported)
+ }
+ val (newCond, newSubPlan, _) = rewriteExistentialExprWithAttrs(conditions, subPlan,
Review Comment:
**Blocking (P1):** This call can receive a positive `InSubquery`, whose SQL result is nullable, but the rewrite replaces it with the non-nullable `exists` attribute. That distinction remains observable inside the parent expression: for `((t2.c1 IN (SELECT col1 FROM t3)) <=> false)` with `t3` containing a non-matching `NULL`, SQL produces `NULL <=> false`, which is false, while this path produces `false <=> false` and admits the `t2` row. The new inner-plan route therefore turns the old attribute-binding failure into silently wrong rows. Please preserve the three-valued result of positive `IN` when synthesizing this existence join.
**Recommended change:** Represent positive nested IN with enough synthesized state to reconstruct TRUE, FALSE, and NULL, and add SQL subquery regressions that place an inner-referencing positive IN under a null-sensitive parent.
**Why this works:** At the positive-IN rewrite owner, preserve both whether any equality condition is true and whether the candidate comparisons can yield unknown when no true match exists, then return a nullable Boolean expression that reconstructs SQL IN semantics. Apply that owner-level representation to both outer- and subquery-plan routes instead of treating an existence bit as the IN value.
**Scope:** Correct the shared nested positive-IN representation and verify three-valued results through the changed inner-side route without altering the accepted both-side rejection policy.
**Compatibility:** The routing decision remains attribute-owner based: inner-only nested subqueries are rewritten on the subquery plan, outer-only/reference-free ones on the outer plan, and correlated both-side ones are rejected.
**Risks:** Multi-column IN equality and correlated join predicates require the unknown signal to follow SQL row-comparison semantics, not merely detect any NULL value on the right side.
**Constraints:** Preserve existing true-match behavior, null-aware NOT IN behavior, attribute ownership on both routing sides, and the structured rejection for correlated both-side subqueries.
**Success:** A positive nested IN returns true for a match, false for no match with no unknown comparison, and null for no match with an unknown comparison, regardless of whether it is routed to the outer or subquery plan. Null-sensitive parents observe the same positive-IN value before and after the existential rewrite and no extra outer rows are admitted. Existing NOT IN null-awareness and correlated both-side rejection remain unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4056570195
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3227,198 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n") {
+ Seq((1), (2), (3), (7)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+
+ // Checks the result, and that every node of the optimized plan can produce the attributes
+ // it references. The latter is what this fix is about: an existence join whose condition
+ // references an attribute produced by neither of its children can still return the right
+ // answer when the nested relation is empty at runtime, because the invalid condition is
+ // then never bound. checkAnswer alone would not catch it, as the missing input checks it
+ // runs only look at the root of the plan.
+ def checkAnswerAndPlan(query: String, expected: Seq[Row]): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ checkAnswer(df, expected)
+ }
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // Same, with a nested subquery that returns no matching row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = 3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(1) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR EXISTS (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is
+ // never true, so only the correlated predicate can be satisfied.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Nil)
+
+ // Without the NULL, c1 NOT IN (3, 9) holds for c1 = 1.
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested subquery that is itself correlated to the query it is nested in.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(1) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // would drop its correlated predicate col1 = c1 and return an extra row, so it is rejected.
+ val correlated =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT 1
+ | FROM t2
+ | WHERE a = c1
+ | OR a IN (SELECT col1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ // The rejection must not depend on whether predicate subqueries in join conditions are
+ // decorrelated, as that rewrite rejects the same shape on its own.
+ Seq("true", "false").foreach { decorrelateInJoinCondition =>
Review Comment:
Correct, and I verified it: with EXISTS as the outer predicate, deleting the new `throw` left the
`true` iteration green, because the arm returns `Project(p.output, join)`, `transformDown` re-offers
that join to the branch at `subquery.scala:249`, and it raises the same condition.
Took your second option and switched the case to NOT IN, whose arm returns a bare `Join` that is
never re-offered. Both iterations now discriminate — deleting the `throw` fails the test on the
first, `true`, iteration. The comment in the test says why NOT IN is used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4056574243
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -450,10 +471,100 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
ExistenceJoin(exists), newConditions, joinHint)
introducedAttrs += exists
exists
+ // A sub-query that `canRewrite` declined is left as it is, children included.
+ case sq @ (_: Exists | Not(_: InSubquery) | _: InSubquery) => sq
+ case other => other.mapChildren(rewrite)
}
}
+ val newExprs = exprs.map(rewrite)
(newExprs.reduceOption(And), newPlan, introducedAttrs.toSeq)
}
+
+ /**
+ * Returns true if `e` references any of the attributes produced by `plan`.
+ */
+ private def referencesPlan(e: Expression, plan: LogicalPlan): Boolean = {
+ e.references.intersect(plan.outputSet).nonEmpty
+ }
+
+ /**
+ * Rewrites the existential sub-queries that are nested in the correlated predicates which
+ * [[PullupCorrelatedPredicates]] hoisted out of a predicate sub-query, and which therefore end
+ * up in the condition of the semi/anti join that replaces that sub-query.
+ *
+ * A hoisted predicate can carry a nested existential sub-query out of the sub-query plan,
+ * because a correlated predicate is hoisted as a whole when it is a disjunction. For example
+ *
+ * SELECT * FROM t1 WHERE EXISTS (
+ * SELECT 1 FROM t2 WHERE t1.a = t2.c1 OR t2.c1 IN (SELECT col1 FROM t3))
+ *
+ * hoists `a = c1 OR c1 IN (SELECT col1 FROM t3)` into the join condition. Such a nested
+ * sub-query must be rewritten against the plan that produces the attributes it references:
+ * the one above references `c1`, which is produced by the sub-query plan and not by the outer
+ * plan, so its existence join has to be built on top of the sub-query plan. Building it on top
+ * of the outer plan instead yields a join whose condition references an attribute that neither
+ * of its children can produce (SPARK-59351).
+ *
+ * A nested sub-query that references both plans cannot be rewritten into an existence join on
+ * either side, so it is left in the join condition, where both plans are in scope. This is only
+ * correct for an uncorrelated sub-query: the join condition of a correlated one is dropped when
+ * it is planned as an in-subquery filter, which would silently change the result, so it is
+ * reported as unsupported instead. Note that such a sub-query can be correlated only to the
+ * sub-query plan, as being correlated to the outer plan as well would require two levels of
+ * correlation, which the Analyzer rejects.
+ *
+ * Returns the rewritten condition along with the updated outer and sub-query plans.
+ */
+ private def rewriteExistentialExprInJoinCondition(
Review Comment:
Added. The scaladoc now says that a subquery left in the join condition is further subject to the
rewrite of predicate subqueries in join conditions, which rejects one referencing both plans under
the default configuration, and that it survives only with
`spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate` disabled. The test that pins
that shape now carries a line saying the conf is set to bypass that branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-5650221572
@cloud-fan @viirya @uros-b Gentle Ping!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4380482961
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
Review Comment:
The scaladoc now separates the two: a carried correlated IN or NOT IN is planned by `PlanSubqueries` as an in-subquery filter that drops its own hoisted condition and returns extra rows, while a carried correlated EXISTS reaches execution as an unevaluable `Exists` and fails. Routing is still right for both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4414370176
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
Review Comment:
Confirmed: the new test reaches the carried both-inputs rejection under both values of decorrelatePredicateSubqueriesInJoinPredicate.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4369134668","thread_id":"inline:4369134668","verdict_sha256":"596d6ef5f193363401f54af770e62e1e49b3bfd27c84efadb74d430a0c108603"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is routed as well, rather than left for that handling to rewrite
+ * under its default configuration: it has no hoisted condition to lose, but it is subject to the
+ * same unknown-result guard as one on the sub-query-plan route, and that handling has no such
+ * guard. Consuming it here is what keeps an IN whose result is observably unknown from becoming
+ * a two-valued `exists` attribute.
+ *
+ * The classification mirrors [[rewriteExistentialExprInSubqueryPlan]], with this join's inputs
+ * in place of the outer and the sub-query plan: a sub-query referencing exactly one input is
+ * routed against it, one referencing both is rejected when it is correlated or carries a
+ * correlated sub-query, since neither input alone can evaluate it, and one whose unknown result
+ * is observable is rejected. A sub-query referencing both inputs or neither and carrying no
+ * correlation is left where it is, as it is at the first level.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ val carried = ArrayBuffer.empty[Expression]
+ collectCarriedExistentialExprs(condition, carried)
+ if (carried.isEmpty) {
+ return join
+ }
+ // A NOT IN is carried as the Not over the IN, which is the unit the rewrite consumes and the
+ // unit the unknown-result collector reports, so both agree on the node below.
+ def negatedIn(sq: Expression): Expression = sq match {
+ case Not(in: InSubquery) => in
+ case other => other
+ }
+ val unknownSensitive = unknownSensitiveInSubqueries(condition, truthOnly = true)
+ def observesUnknown(sq: Expression): Boolean = unknownSensitive.exists(_.fastEquals(sq))
+ def referencesInput(sq: Expression, input: LogicalPlan): Boolean =
+ effectiveReferences(sq).exists(input.outputSet.contains)
+ def referencesBothInputs(sq: Expression): Boolean =
+ referencesInput(sq, join.left) && referencesInput(sq, join.right)
+ def referencesOneInput(sq: Expression): Boolean =
+ referencesInput(sq, join.left) != referencesInput(sq, join.right)
+
+ // The two rejections of the sub-query-plan route apply here for the same reasons, against the
+ // inputs of this join rather than against the outer and the sub-query plan, and each raises
+ // the condition whose message describes it.
+ val referencingBothInputs = carried.filter { sq =>
+ referencesBothInputs(sq) &&
+ (hasCorrelatedCondition(sq) || carriesCorrelatedSubquery(negatedIn(sq)))
+ }
+ if (referencingBothInputs.nonEmpty) {
+ throw QueryCompilationErrors.nestedSubqueryReferencingOuterAndInnerQueryError(
+ referencingBothInputs.toSeq)
+ }
+ val unknownResult = carried.filter(sq => referencesOneInput(sq) && observesUnknown(sq))
+ if (unknownResult.nonEmpty) {
+ throw QueryCompilationErrors.nestedInSubqueryWithUnknownResultError(unknownResult.toSeq)
+ }
+ var newCondition = condition
+ def route(plan: LogicalPlan, isInput: Expression => Boolean): LogicalPlan = {
+ carried.filter(sq => referencesOneInput(sq) && isInput(sq)).foldLeft(plan) {
+ case (p, sq) =>
+ val (rewritten, newInput, _) = rewriteExistentialExprWithAttrs(Seq(sq), p)
+ newCondition = newCondition.transform {
Review Comment:
Thanks, routing NOT IN units first fixes this repro, but it is not equivalent to substituting at the collected position. On a pre-existing route the router keeps an uncorrelated NOT IN whose unknown is observable, e.g. `(NOT s.b) = s2.f`, yet still routes the bare `s.b`. The bare unit's substitution then rewrites the IN inside the kept NOT, so with either join-condition flag off -1 is returned again where the merge base returned only 1. Substituting each unit only where it was collected covers this case as well.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4369134655","thread_id":"inline:4369134655","verdict_sha256":"596d6ef5f193363401f54af770e62e1e49b3bfd27c84efadb74d430a0c108603"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,615 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
Review Comment:
Thanks, the IN and EXISTS failure modes are now separated. The same sentence still says nothing else reaches a carried sub-query, which this review raises as a separate point.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4369134661","thread_id":"inline:4369134661","verdict_sha256":"596d6ef5f193363401f54af770e62e1e49b3bfd27c84efadb74d430a0c108603"} -->
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -184,8 +188,20 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
// Use EXISTS if performance matters to you.
// Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
+ val dedupSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
+ val inConditions = values.zip(dedupSub.output).map(EqualTo.tupled)
+ // The hoisted correlated predicates may carry existential sub-queries that only the
+ // sub-query plan can evaluate, see rewriteExistentialExprInJoinCondition.
+ // This arm returns a bare Join, which this rule does not offer to its own handling of
Review Comment:
Thanks, the production comment now states the last-conjunct condition. Its explanation of which later conjuncts expose this anti join is incomplete, which this review raises.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4346076714","thread_id":"inline:4346076714","verdict_sha256":"596d6ef5f193363401f54af770e62e1e49b3bfd27c84efadb74d430a0c108603"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4356960955
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:
##########
@@ -398,62 +414,568 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
(newExpr, newPlan)
}
+ /**
+ * Rewrites the existential sub-queries carried by the condition of an existence join that this
+ * rule has just built, against whichever of that join's own inputs can evaluate them.
+ *
+ * [[rewriteExistentialExprWithAttrs]] grafts the condition that [[PullupCorrelatedPredicates]]
+ * hoisted into a nested sub-query onto the existence join it builds for that sub-query, and
+ * replaces the sub-query atomically without descending into what the condition carries. A
+ * correlated existential sub-query carried along references the plan of the sub-query it was
+ * hoisted out of, which is an input of the join just built and of no plan seen before it, so
+ * nothing else reaches it: left there, `PlanSubqueries` plans it as an in-subquery filter and
+ * drops its own hoisted condition, which returns extra rows.
+ *
+ * This is what the handling of predicate sub-queries in join conditions does for a join the user
+ * wrote, and doing it here too makes the result independent of
+ * `spark.sql.optimizer.decorrelatePredicateSubqueriesInJoinPredicate.enabled`, which gates that
+ * handling. The routing recurses, since the joins built here go through it in turn.
+ *
+ * An uncorrelated sub-query is deliberately left where it is. It has no hoisted condition to
+ * lose, and `InSubqueryExec` evaluates it with the three-valued semantics that the `exists`
+ * attribute of an existence join cannot carry.
+ */
+ private def rewriteCarriedExistentialExprs(join: Join): LogicalPlan = {
+ if (!join.condition.exists(_.containsAnyPattern(EXISTS_SUBQUERY, IN_SUBQUERY))) {
+ return join
+ }
+ val condition = join.condition.get
+ // Only a correlated sub-query has to be consumed here, see above.
+ val carried = condition.collect {
+ case sq @ (_: Exists | _: InSubquery) if hasCorrelatedCondition(sq) => sq
Review Comment:
Confirmed, and this one undoes the claim I made last round: the two configurations did **not** agree on this shape, only on the correlated one I had tested.
| | result |
|---|---|
| before, `conf=true` (default) | `1, 2, 3` |
| before, `conf=false` | `1` |
| **now, both** | **rejected, `NESTED_IN_SUBQUERY_WITH_UNKNOWN_RESULT`** |
| DuckDB | `1` |
My premise that `InSubqueryExec` would evaluate an uncorrelated carried IN three-valued was wrong on this route, for exactly the reason you give: the top-down pass reaches the existence join in the `case j: Join` arm, which has no unknown guard, and two-values it under the default configuration.
The router now classifies carried sub-queries with the same rules as `rewriteExistentialExprInSubqueryPlan`, against this join's inputs instead of the outer and the sub-query plan: uncorrelated ones are considered too, one referencing exactly one input is routed, one referencing both is rejected when correlated or carrying a correlated sub-query, and one referencing both or neither with no correlation is left in place. The scaladoc no longer says an uncorrelated carried sub-query is left for `InSubqueryExec`; it now says why consuming it here is what keeps it under the unknown guard.
On the trade-off you flagged: rejecting an uncorrelated unknown-sensitive carried IN under both configurations does turn a query that `conf=false` answers correctly today into an error. I have taken that, since it matches the first-level policy and leaves no shape where the two configurations disagree. The truth-only form of the same query is still rewritten and still returns `1` under both settings, which the new test asserts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "pratham76 (via GitHub)" <gi...@apache.org>.
pratham76 commented on PR #58656:
URL: https://github.com/apache/spark/pull/58656#issuecomment-6098592689
@cloud-fan Gentle Ping! Could you PTAL on the addressed comments? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59351][SQL] Rewrite nested subqueries in hoisted correlated predicates against the subquery plan [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #58656:
URL: https://github.com/apache/spark/pull/58656#discussion_r4346082541
##########
sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala:
##########
@@ -3227,4 +3224,540 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ /**
+ * Checks the result of `query`, that every node of its optimized plan can produce the attributes
+ * it references, and that the existence join a nested subquery was rewritten into sits on the
+ * subquery side of the semi/anti join.
+ *
+ * The plan checks are what SPARK-59351 is about: an existence join whose condition references an
+ * attribute produced by neither of its children can still return the right answer when the
+ * nested relation is empty at runtime, because the invalid condition is then never bound, and a
+ * rewrite that placed the join on the other side would return the right answer here too. Note
+ * that missingInput cannot see a subquery expression that survives in a condition, since
+ * QueryPlan.references folds only the outer attributes of one and collect does not walk its
+ * plan, so it is not a substitute for asserting the shape.
+ */
+ private def checkAnswerAndPlan(
+ query: String,
+ expected: Seq[Row],
+ existenceJoinOnSubqueryPlan: Boolean = true): Unit = {
+ val df = sql(query)
+ val plan = df.queryExecution.optimizedPlan
+ val invalidNodes = plan.collect { case p if p.missingInput.nonEmpty => p }
+ assert(invalidNodes.isEmpty,
+ s"""Plan nodes reference non-reachable attributes:
+ |${invalidNodes.mkString("\n")}
+ |$plan""".stripMargin)
+ if (existenceJoinOnSubqueryPlan) {
+ val semiOrAnti = plan.collectFirst {
+ case j: Join if j.joinType == LeftSemi || j.joinType == LeftAnti => j
+ }
+ assert(semiOrAnti.nonEmpty, s"expected a semi or anti join in\n$plan")
+ val existenceJoins = plan.collect {
+ case j: Join if j.joinType.isInstanceOf[ExistenceJoin] => j
+ }
+ assert(existenceJoins.nonEmpty, s"expected an existence join in\n$plan")
+ // The existence join must be under the right side of the semi/anti join, which is the
+ // subquery plan, and not under its left side, which is the outer plan.
+ assert(existenceJoins.forall(j => semiOrAnti.get.right.exists(_.fastEquals(j))),
+ s"the existence join must sit on the subquery side in\n$plan")
+ }
+ checkAnswer(df, expected)
+ }
+
+ test("SPARK-59351: nested subquery referencing the inner query becomes an existence join") {
+ // SPARK-45580 covers the case where the nested subquery references the outer query, in which
+ // case its existence join is built on top of the outer plan. Here the nested subquery
+ // references the query it is nested in, so the existence join has to be built on top of the
+ // subquery plan instead.
+ withTempView("t1", "t2", "t3", "t3n", "t4", "t5") {
+ Seq((1), (2), (3), (7), (9)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ // t3 shares a value with t2, t5 does not, so the nested subquery decides the answer below.
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ Seq((3), (7)).toDF("col1").persist().createOrReplaceTempView("t5")
+ Seq(Some(3), Some(9), None).toDF("col1").persist().createOrReplaceTempView("t3n")
+ // A correlated nested IN over t4 is false for every c1 in t2, while the same IN without its
+ // correlated predicate is true for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // EXISTS rewritten as a left semi join. The correlated predicate is a disjunction, so it
+ // is pulled up as a whole and carries the nested IN-subquery, which references c1, out of
+ // the subquery plan.
+ val query1 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query1, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // Same over t5, which shares no value with t2, so the nested IN is false for every c1 and
+ // only the correlated predicate can hold: a mistake making it true would return every row.
+ val query2 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query2, Row(1) :: Row(9) :: Nil)
+
+ // NOT EXISTS rewritten as a left anti join.
+ val query3 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE NOT EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 IN (SELECT col1 FROM t5)
+ |)""".stripMargin
+ checkAnswerAndPlan(query3, Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // IN-subquery rewritten as a left semi join. The hoisted predicate is a > c1 rather than
+ // the key equality a = c1, so the answer is a IN (t2 INTERSECT t3) and depends on the
+ // nested subquery: dropping it would leave no row at all.
+ val query4 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query4, Row(9) :: Nil)
+
+ // NOT IN-subquery rewritten as a null-aware left anti join, with a nested EXISTS. Only the
+ // nested EXISTS keeps 9 out of the answer, as a > c1 alone does not hold for it.
+ val query5 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE a NOT IN (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR EXISTS (SELECT 1 FROM t3 WHERE col1 = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query5, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Nil)
+
+ // A nested NOT IN-subquery keeps its null-aware semantics: c1 NOT IN (3, 9, NULL) is never
+ // true, so only the correlated predicate can be satisfied. Compare with query7, where the
+ // same NOT IN over a relation without NULL holds for c1 = 1 and returns every row.
+ val query6 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3n)
+ |)""".stripMargin
+ checkAnswerAndPlan(query6, Row(1) :: Row(9) :: Nil)
+
+ val query7 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a = c1
+ | OR c1 NOT IN (SELECT col1 FROM t3)
+ |)""".stripMargin
+ checkAnswerAndPlan(query7, Row(1) :: Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+
+ // A nested subquery correlated to the query it is nested in, on a column it does not
+ // project, so its correlated predicate changes the answer: without it the nested IN would
+ // hold for c1 = 1 and row 1 would be returned as well.
+ val query8 =
+ """
+ |SELECT *
+ |FROM t1
+ |WHERE EXISTS (
+ | SELECT c1
+ | FROM t2
+ | WHERE a > c1
+ | OR c1 IN (SELECT col1 FROM t4 WHERE k = c1)
+ |)""".stripMargin
+ checkAnswerAndPlan(query8, Row(2) :: Row(3) :: Row(7) :: Row(9) :: Nil)
+ }
+ }
+
+ test("SPARK-59351: nested subquery referencing both the outer and the inner query") {
+ withTempView("t1", "t2", "t3", "t4") {
+ Seq((1), (2), (3)).toDF("a").persist().createOrReplaceTempView("t1")
+ Seq((1), (8), (9)).toDF("c1").persist().createOrReplaceTempView("t2")
+ Seq((3), (9)).toDF("col1").persist().createOrReplaceTempView("t3")
+ // Correlated, this is false for every c1 in t2; without its correlated predicate it holds
+ // for 1 and 9, so the correlation decides the answer below.
+ Seq((1, 9), (9, 1)).toDF("col1", "k").persist().createOrReplaceTempView("t4")
+
+ // The nested subquery references the outer query through its values and the inner query
+ // through its own correlated predicate, so it can be rewritten into an existence join on
+ // neither side and stays in the join condition. Planning it there as an in-subquery filter
+ // drops its correlated predicate col1 = c1, which is why it is rejected rather than left.
+ // These rows do not distinguish the two forms; the EXISTS form of this shape does, and
+ // returned an extra row before the rejection was added.
+ //
+ // NOT IN is the outer predicate on purpose: its rewrite returns a bare Join, which
+ // RewritePredicateSubquery does not offer to its own handling of predicate subqueries in
+ // join conditions, so this rejection is the only one that can fire. With EXISTS the rewrite
+ // returns a Project over the join, which that handling then rejects on its own under the
+ // default configuration, and the assertion would hold either way.
Review Comment:
Thanks. The new wording is right only if the new throw were removed: as written it says the EXISTS form is rejected by the join-condition handling with UNSUPPORTED_CORRELATED_EXPRESSION_IN_JOIN_CONDITION, but today it fails earlier with NESTED_SUBQUERY_REFERENCING_OUTER_AND_INNER_QUERY; see the comment on SubquerySuite.scala.
<!-- SPARK_DEV_REVIEW_REPLY {"feedback_id":"inline:4281437548","thread_id":"inline:4281437548","verdict_sha256":"3bb54f2a3a4b20553e9d3ca5caee9c4a8fc61921fc05f7c2363e10d0f4e5e7b0"} -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org