You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (Jira)" <ji...@apache.org> on 2019/10/04 18:21:01 UTC
[jira] [Updated] (SPARK-11150) Dynamic partition pruning
[ https://issues.apache.org/jira/browse/SPARK-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li updated SPARK-11150:
----------------------------
Description:
Implements dynamic partition pruning by adding a dynamic-partition-pruning filter if there is a partitioned table and a filter on the dimension table. The filter is then planned using a heuristic approach:
# As a broadcast relation if it is a broadcast hash join. The broadcast relation will then be transformed into a reused broadcast exchange by the {{ReuseExchange}} rule; or
# As a subquery duplicate if the estimated benefit of partition table scan being saved is greater than the estimated cost of the extra scan of the duplicated subquery; otherwise
# As a bypassed condition ({{true}}).
!image-2019-10-04-11-20-02-616.png!
was:
Partitions are not pruned when joined on the partition columns.
This is the same issue as HIVE-9152.
Ex:
Select .... from tab where partcol=1 will prune on value 1
Select .... from tab join dim on (dim.partcol=tab.partcol) where dim.partcol=1 will scan all partitions.
Tables are based on parquets.
> Dynamic partition pruning
> -------------------------
>
> Key: SPARK-11150
> URL: https://issues.apache.org/jira/browse/SPARK-11150
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 1.5.1, 1.6.0, 2.0.0, 2.1.2, 2.2.1, 2.3.0
> Reporter: Younes
> Assignee: Wei Xue
> Priority: Major
> Fix For: 3.0.0
>
> Attachments: image-2019-10-04-11-20-02-616.png
>
>
> Implements dynamic partition pruning by adding a dynamic-partition-pruning filter if there is a partitioned table and a filter on the dimension table. The filter is then planned using a heuristic approach:
> # As a broadcast relation if it is a broadcast hash join. The broadcast relation will then be transformed into a reused broadcast exchange by the {{ReuseExchange}} rule; or
> # As a subquery duplicate if the estimated benefit of partition table scan being saved is greater than the estimated cost of the extra scan of the duplicated subquery; otherwise
> # As a bypassed condition ({{true}}).
>
> !image-2019-10-04-11-20-02-616.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org