You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (JIRA)" <ji...@apache.org> on 2018/01/31 17:15:00 UTC
[jira] [Updated] (SPARK-20331) Broaden support for Hive partition
pruning predicate pushdown
[ https://issues.apache.org/jira/browse/SPARK-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li updated SPARK-20331:
----------------------------
Labels: releasenotes (was: )
> Broaden support for Hive partition pruning predicate pushdown
> -------------------------------------------------------------
>
> Key: SPARK-20331
> URL: https://issues.apache.org/jira/browse/SPARK-20331
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Michael Allman
> Assignee: Michael Allman
> Priority: Major
> Labels: releasenotes
> Fix For: 2.3.0
>
>
> Spark 2.1 introduced scalable support for Hive tables with huge numbers of partitions. Key to leveraging this support is the ability to prune unnecessary table partitions to answer queries. Spark supports a subset of the class of partition pruning predicates that the Hive metastore supports. If a user writes a query with a partition pruning predicate that is *not* supported by Spark, Spark falls back to loading all partitions and pruning client-side. We want to broaden Spark's current partition pruning predicate pushdown capabilities.
> One of the key missing capabilities is support for disjunctions. For example, for a table partitioned by date, specifying with a predicate like
> {code}date = 20161011 or date = 20161014{code}
> will result in Spark fetching all partitions. For a table partitioned by date and hour, querying a range of hours across dates can be quite difficult to accomplish without fetching all partition metadata.
> The current partition pruning support supports only comparisons against literals. We can expand that to foldable expressions by evaluating them at planning time.
> We can also implement support for the "IN" comparison by expanding it to a sequence of "OR"s.
> This ticket covers those enhancements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org