You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Szehon Ho (Jira)" <ji...@apache.org> on 2026/05/29 00:40:00 UTC
[jira] [Updated] (SPARK-53928) Enhance DSV2 partition filtering using catalyst expression
[ https://issues.apache.org/jira/browse/SPARK-53928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Szehon Ho updated SPARK-53928:
------------------------------
Parent: SPARK-55596
Issue Type: Sub-task (was: Improvement)
> Enhance DSV2 partition filtering using catalyst expression
> ----------------------------------------------------------
>
> Key: SPARK-53928
> URL: https://issues.apache.org/jira/browse/SPARK-53928
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: Szehon Ho
> Priority: Major
> Labels: pull-request-available
>
> Currently, Spark converts Catalyst Expression to either Filter or Predicate and pushes it to DSV2 via SupportsPushdownFilters and SupportsPushdownV2Filters API's.
> However, some Spark filters may not convert cleanly. For example, trim(part_col) = 'a'. There are cases where DSV2 can return the exact partition value(s) to spark for its InputPartition, and Spark can use the original catalyst expression for filtering.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org