[GitHub] [spark] cloud-fan commented on pull request #38464: [SPARK-32628][SQL] Use bloom filter to improve dynamic partition pruning

[GitBox <gi...@apache.org>]

cloud-fan commented on PR #38464:
URL: https://github.com/apache/spark/pull/38464#issuecomment-1316611596

   I agree with using bloom filters, as the size estimation can be wrong and the build size can be too large that `InSubquery` can't work. However, this PR contains another optimization that forces shuffle reuse when building the subquery to build bloom filter. Can we do it later with more discussions? This is a general optimization that can apply in other places as well: InSubquery DPP, bloom filter join.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org