[jira] [Created] (SPARK-28375) Fix PullupCorrelatedPredicates optimizer rule to enforce idempotence

["Yesheng Ma (JIRA)" <ji...@apache.org>]

Yesheng Ma created SPARK-28375:
----------------------------------

             Summary: Fix PullupCorrelatedPredicates optimizer rule to enforce idempotence
                 Key: SPARK-28375
                 URL: https://issues.apache.org/jira/browse/SPARK-28375
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: Yesheng Ma


The current PullupCorrelatedPredicates implementation can accidentally remove predicates for multiple runs.

For example, for the following logical plan, one more optimizer run can remove the predicate in the SubqueryExpresssion.
{code:java}
# Optimized
Project [a#0]
+- Filter a#0 IN (list#4 [(b#1 < d#3)])
   :  +- Project [c#2, d#3]
   :     +- LocalRelation <empty>, [c#2, d#3]
   +- LocalRelation <empty>, [a#0, b#1]

# Double optimized
Project [a#0]
+- Filter a#0 IN (list#4 [])
   :  +- Project [c#2, d#3]
   :     +- LocalRelation <empty>, [c#2, d#3]
   +- LocalRelation <empty>, [a#0, b#1]
{code}
 

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org