[jira] [Created] (SPARK-25127) DataSourceV2: Remove SupportsPushDownCatalystFilters

["Ryan Blue (JIRA)" <ji...@apache.org>]

Ryan Blue created SPARK-25127:
---------------------------------

             Summary: DataSourceV2: Remove SupportsPushDownCatalystFilters
                 Key: SPARK-25127
                 URL: https://issues.apache.org/jira/browse/SPARK-25127
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 2.3.1, 2.3.0, 2.4.0
            Reporter: Ryan Blue


Discussion about adding TableCatalog on the dev list focused around whether Expression should be used in the public DataSourceV2 API, with SupportsPushDownCatalystFilters as an example of where it is already exposed. The early consensus is that Expression should not be exposed in the public API.

From [~rxin]:

bq. I completely disagree with using Expression in critical public APIs that we expect a lot of developers to use . . . If we are depending on Expressions on the more common APIs in dsv2 already, we should revisit that.

The main use of this API is to pass Expression to FileFormat classes that used Expression instead of Filter. External sources also use it for more complex push-down, like {{to_date(ts) = '2018-05-13'}}, but those uses can be done with Analyzer rules or when translating to Filters.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org