You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Penglei Shi (Jira)" <ji...@apache.org> on 2021/10/29 03:12:00 UTC
[jira] [Commented] (SPARK-32268) Bloom Filter Join
[ https://issues.apache.org/jira/browse/SPARK-32268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17435755#comment-17435755 ]
Penglei Shi commented on SPARK-32268:
-------------------------------------
Hi [~yumwang], I have some problems about bloom filter in my test(tpcds-q37),
# For computing left and right rowCount of the join, the estimated size of the join(inventory, item, date_dim) is huge, because it's a product of children size according to SizeInBytesOnlyStatsPlanVisitor.visitJoin, this makes a wrong small side.
# For the configuration `spark.sql.optimizer.dynamicPartitionPruning.pruningSideExtraFilterRatio`, the default value is 0.04, it looks like the filtering side row count should less than the pruning size row count * 0.04 * 0.04 in `pruningHasBenefit`, this also makes q37 not working with bloom filter.
Could you help me?
> Bloom Filter Join
> -----------------
>
> Key: SPARK-32268
> URL: https://issues.apache.org/jira/browse/SPARK-32268
> Project: Spark
> Issue Type: New Feature
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Yuming Wang
> Assignee: Yuming Wang
> Priority: Major
> Attachments: q16-bloom-filter.jpg, q16-default.jpg
>
>
> We can improve the performance of some joins by pre-filtering one side of a join using a Bloom filter and IN predicate generated from the values from the other side of the join.
> For example:[tpcds/q16.sql|https://github.com/apache/spark/blob/a78d6ce376edf2a8836e01f47b9dff5371058d4c/sql/core/src/test/resources/tpcds/q16.sql]. [Before this optimization|https://issues.apache.org/jira/secure/attachment/13007418/q16-default.jpg]. [After this optimization|https://issues.apache.org/jira/secure/attachment/13007416/q16-bloom-filter.jpg].
> *Query Performance Benchmarks: TPC-DS Performance Evaluation*
> Our setup for running TPC-DS benchmark was as follows: TPC-DS 5T and Partitioned Parquet table
>
> |Query|Default(Seconds)|Enable Bloom Filter Join(Seconds)|
> |tpcds q16|84|46|
> |tpcds q36|29|21|
> |tpcds q57|39|28|
> |tpcds q94|42|34|
> |tpcds q95|306|288|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org