[spark] branch master updated: [SPARK-39377][SQL][TESTS] Normalize expr ids in ListQuery and Exists expressions

[yu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yumwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new cc0bf563b8c [SPARK-39377][SQL][TESTS] Normalize expr ids in ListQuery and Exists expressions
cc0bf563b8c is described below

commit cc0bf563b8caea21da5692f05e34b5f77e002ab9
Author: Yuming Wang <yu...@ebay.com>
AuthorDate: Mon Jun 6 12:18:18 2022 +0800

    [SPARK-39377][SQL][TESTS] Normalize expr ids in ListQuery and Exists expressions
    
    ### What changes were proposed in this pull request?
    
    This PR makes it normalize expr ids in `ListQuery` and `Exists` expressions. For example:
    ```scala
    val testRelation = LocalRelation($"a".int, $"b".int, $"c".int)
    val x = testRelation.as("x")
    val y = testRelation.as("y")
    
    val originalQuery = x.where($"x.a".in(ListQuery(y.select($"b")))).analyze
    
    println(Optimize.execute(normalizeExprIds(originalQuery)))
    ```
    Before this PR:
    ```
    Filter a#0 IN (list#0 [])
    :  +- Project [b#15]
    :     +- SubqueryAlias y
    :        +- LocalRelation <empty>, [a#14, b#15, c#16]
    +- LocalRelation <empty>, [a#0, b#0, c#0]
    ```
    After this PR:
    ```
    Filter a#0 IN (list#0 [])
    :  +- Project [b#0]
    :     +- SubqueryAlias y
    :        +- LocalRelation <empty>, [a#0, b#0, c#0]
    +- LocalRelation <empty>, [a#0, b#0, c#0]
    ```
    
    ### Why are the changes needed?
    
    `PlanTestBase.comparePlans` fails in some cases because the expr id not equal.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    manual test.
    
    Closes #36764 from wangyum/SPARK-39377.
    
    Authored-by: Yuming Wang <yu...@ebay.com>
    Signed-off-by: Yuming Wang <yu...@ebay.com>
---
 .../test/scala/org/apache/spark/sql/catalyst/plans/PlanTest.scala  | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/PlanTest.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/PlanTest.scala
index c14b2189394..2934f97a89e 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/PlanTest.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/PlanTest.scala
@@ -76,9 +76,12 @@ trait PlanTestBase extends PredicateHelper with SQLHelper with SQLConfHelper { s
       case s: LateralSubquery =>
         s.copy(plan = normalizeExprIds(s.plan), exprId = ExprId(0))
       case e: Exists =>
-        e.copy(exprId = ExprId(0))
+        e.copy(plan = normalizeExprIds(e.plan), exprId = ExprId(0))
       case l: ListQuery =>
-        l.copy(exprId = ExprId(0))
+        l.copy(
+          plan = normalizeExprIds(l.plan),
+          exprId = ExprId(0),
+          childOutputs = l.childOutputs.map(_.withExprId(ExprId(0))))
       case a: AttributeReference =>
         AttributeReference(a.name, a.dataType, a.nullable)(exprId = ExprId(0))
       case OuterReference(a: AttributeReference) =>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org