You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Liang-Chi Hsieh (JIRA)" <ji...@apache.org> on 2015/09/26 09:51:04 UTC

[jira] [Created] (SPARK-10841) Add pushdown support of UDF for parquet

Liang-Chi Hsieh created SPARK-10841:
---------------------------------------

             Summary: Add pushdown support of UDF for parquet
                 Key: SPARK-10841
                 URL: https://issues.apache.org/jira/browse/SPARK-10841
             Project: Spark
          Issue Type: New Feature
          Components: SQL
            Reporter: Liang-Chi Hsieh


JIRA: 

Currently we can't push down filters involving UDFs to Parquet. In practice, we have some usage of UDFs in filters, e.g.,

     SELECT * FROM table WHERE udf(customer_id) = "ABC"

In above query, `customer_id` is a column storing customer id in some way. `udf` is a function used to parse this column to string value. Without pushing down the filter to Parquet, we will fetch all data from many Parquet files and then perform filtering in Spark.

Using Parquet's `UserDefinedPredicate`, we can push down these filters to Parquet. This patch adds the support for this. This patch currently only implements `EqualTo` predicate. Other predicates such as `LessThan`, `GreaterThan` will be implemented in following PRs.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org