[jira] [Created] (SPARK-21759) PullupCorrelatedPredicates can produce unresolved plan

["Liang-Chi Hsieh (JIRA)" <ji...@apache.org>]

Liang-Chi Hsieh created SPARK-21759:
---------------------------------------

             Summary: PullupCorrelatedPredicates can produce unresolved plan
                 Key: SPARK-21759
                 URL: https://issues.apache.org/jira/browse/SPARK-21759
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.2.0
            Reporter: Liang-Chi Hsieh


With the check for structural integrity proposed in SPARK-21726, I found that an optimization rule {{PullupCorrelatedPredicates}} can produce unresolved plans.

For a correlated IN query like:
{code}
Project [a#0]
+- Filter a#0 IN (list#4 [b#1])
   :  +- Project [c#2]
   :     +- Filter (outer(b#1) < d#3)
   :        +- LocalRelation <empty>, [c#2, d#3]
   +- LocalRelation <empty>, [a#0, b#1]
{code}

After {{PullupCorrelatedPredicates}}, it produces query plan like:
{code}
'Project [a#0]
+- 'Filter a#0 IN (list#4 [(b#1 < d#3)])
   :  +- Project [c#2, d#3]
   :     +- LocalRelation <empty>, [c#2, d#3]
   +- LocalRelation <empty>, [a#0, b#1]
{code}

Because the correlated predicate involves another attribute {{d#3}} in subquery, it has been pulled out and added into the {{Project}} on the top of the subquery.

When {{list}} in {{In}} contains just one {{ListQuery}}, {{In.checkInputDataTypes}} checks if the size of {{value}} expressions matches the output size of subquery. In the above example, there is only {{value}} expression and the subquery output has two attributes {{c#2, d#3}}, so it fails the check and {{In.resolved}} returns {{false}}.

We should let {{PullupCorrelatedPredicates}} produce resolved plans to pass the structural integrity check.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org