You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Szehon Ho (Jira)" <ji...@apache.org> on 2025/10/15 22:36:00 UTC

[jira] [Created] (SPARK-53928) Enhance DSV2 partition filtering using catalyst expression

Szehon Ho created SPARK-53928:
---------------------------------

             Summary: Enhance DSV2 partition filtering using catalyst expression
                 Key: SPARK-53928
                 URL: https://issues.apache.org/jira/browse/SPARK-53928
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.1.0
            Reporter: Szehon Ho


Currently, Spark converts Catalyst Expression to either Filter or Predicate and pushes it to DSV2 via SupportsPushdownFilters and SupportsPushdownV2Filters API's.

However, some Spark filters may not convert cleanly.  For example, trim(part_col) = 'a'.  There are cases where DSV2 can return the exact partition value(s) to spark for its InputPartition, and Spark can use the original catalyst expression for filtering.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org