You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Chao Sun (Jira)" <ji...@apache.org> on 2023/09/15 17:19:00 UTC

[jira] [Resolved] (SPARK-36695) Allow passing V2 functions to data sources via V2 filters

     [ https://issues.apache.org/jira/browse/SPARK-36695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chao Sun resolved SPARK-36695.
------------------------------
    Resolution: Duplicate

> Allow passing V2 functions to data sources via V2 filters
> ---------------------------------------------------------
>
>                 Key: SPARK-36695
>                 URL: https://issues.apache.org/jira/browse/SPARK-36695
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 3.3.0
>            Reporter: Chao Sun
>            Priority: Major
>
> The V2 filter API currently only allow {{NamedReference}} in predicates that are pushed down to data sources. It may be beneficial to allow V2 functions in predicates as well so that we can implement function pushdown. This feature is also supported by Trino (Presto).
> One use case is we can pushdown predicates such as {{bucket(col, 32) = 10}} which will allow data sources such as Iceberg to only scan a single partition.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org