You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/07/22 05:59:24 UTC

[GitHub] [spark] jovany-wang edited a comment on pull request #29065: [WIP][SPARK-32268][SQL] Bloom Filter Join

jovany-wang edited a comment on pull request #29065:
URL: https://github.com/apache/spark/pull/29065#issuecomment-662255897

Hi @wangyum , This is a nice PR to me. But some issues in my mind should be thrown here.

I didn't do more perf between MinMax and Bloom, but in my personal sense, these may effect the perf of different cases.
So how about making these things more general? like:
```
DynamicFilter
|
Is the filtering key partitioned?
/ \
Y N
/ \
DPP filter Choose a best filter for it. (from MinMax, Bloom or other filters such as index filter, etc)
Note: Not all of the filters can be pushed to scan.
```
That is just a rough idea, but the key point is to make DynamicFilter(or name it RuntimeFilter) more general(that means both of MinMaxFilter, BloomFilter and DPPFilter are DynamicFilter), so that it will be easy to get extended.

I have seen another proposal about RuntimeFilter(MinMax) before, so making things easy to be extended should be important as well as the perf result. em, maybe it's hard to make it more extendable.

Feel free to point my incorrect understanding out, thx.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org