[jira] [Created] (SPARK-45621) Add feature to evaluate subquery before Optimizer rule to push down filter

["Maytas Monsereenusorn (Jira)" <ji...@apache.org>]

Maytas Monsereenusorn created SPARK-45621:
---------------------------------------------

             Summary: Add feature to evaluate subquery before Optimizer rule to push down filter
                 Key: SPARK-45621
                 URL: https://issues.apache.org/jira/browse/SPARK-45621
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 3.3.2
            Reporter: Maytas Monsereenusorn


Some queries can benefit from having it's scalar subquery in the filter evaluated while planning so that the scalar result (from the subquery) can be push down. 

This adds a new feature(which is disabled by default to maintain current behavior) that would evaluate scalar subqueries in the Optimizer before rule to push down filter. 

For example, a query like 
{code:java}
select * from t2 where b > (select max(a) from t1) {code}
where t1 is a small table but t2 is a very large table can benefit if we first evaluate the subquery then push down the result to the pushed filter (instead of having the subquery in the post scan filter)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org