[jira] [Commented] (SPARK-5777) Completes data source filter types and remove CatalystScan

["Michael Armbrust (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-5777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14549111#comment-14549111 ] 

Michael Armbrust commented on SPARK-5777:
-----------------------------------------

I agree that this shouldn't be consider public API, but is there a problem leaving it in for experimentation (similar to {{sqlContext.experimental.extraStrategies}})?

> Completes data source filter types and remove CatalystScan
> ----------------------------------------------------------
>
>                 Key: SPARK-5777
>                 URL: https://issues.apache.org/jira/browse/SPARK-5777
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 1.2.0, 1.2.1, 1.3.0
>            Reporter: Cheng Lian
>            Assignee: Cheng Lian
>
> Currently the data source API only supports a fraction of common filters, esp. {{And}} is not supported yet. To workaround this issue and enable full filter push-down optimization in the Parquet data source, {{CatalystScan}} was introduced to receive full Catalyst filter expressions. This class should be removed, since in principle, data source implementations shouldn't touch Catalyst expressions (which are not part of the public developer API).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org