[jira] [Updated] (SPARK-45621) Add feature to evaluate subquery before push down filter Optimizer rule

["Maytas Monsereenusorn (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-45621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Maytas Monsereenusorn updated SPARK-45621:
------------------------------------------
    Summary: Add feature to evaluate subquery before push down filter Optimizer rule  (was: Add feature to evaluate subquery before Optimizer rule to push down filter)

> Add feature to evaluate subquery before push down filter Optimizer rule
> -----------------------------------------------------------------------
>
>                 Key: SPARK-45621
>                 URL: https://issues.apache.org/jira/browse/SPARK-45621
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 3.3.2
>            Reporter: Maytas Monsereenusorn
>            Priority: Major
>
> Some queries can benefit from having it's scalar subquery in the filter evaluated while planning so that the scalar result (from the subquery) can be push down. 
> This adds a new feature(which is disabled by default to maintain current behavior) that would evaluate scalar subqueries in the Optimizer before rule to push down filter. 
> For example, a query like 
> {code:java}
> select * from t2 where b > (select max(a) from t1) {code}
> where t1 is a small table but t2 is a very large table can benefit if we first evaluate the subquery then push down the result to the pushed filter (instead of having the subquery in the post scan filter)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org