You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Szehon Ho (Jira)" <ji...@apache.org> on 2025/10/15 22:36:00 UTC
[jira] [Created] (SPARK-53928) Enhance DSV2 partition filtering using catalyst expression
Szehon Ho created SPARK-53928:
---------------------------------
Summary: Enhance DSV2 partition filtering using catalyst expression
Key: SPARK-53928
URL: https://issues.apache.org/jira/browse/SPARK-53928
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: Szehon Ho
Currently, Spark converts Catalyst Expression to either Filter or Predicate and pushes it to DSV2 via SupportsPushdownFilters and SupportsPushdownV2Filters API's.
However, some Spark filters may not convert cleanly. For example, trim(part_col) = 'a'. There are cases where DSV2 can return the exact partition value(s) to spark for its InputPartition, and Spark can use the original catalyst expression for filtering.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org