You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/07 08:24:36 UTC
[GitHub] [spark] jaceklaskowski commented on a change in pull request #33914: [SPARK-32268][SQL] Dynamic bloom filter join pruning
jaceklaskowski commented on a change in pull request #33914:
URL: https://github.com/apache/spark/pull/33914#discussion_r703286889
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -353,6 +353,23 @@ object SQLConf {
.checkValue(ratio => ratio > 0.0 && ratio <= 1.0, "The ratio value must be in (0.0, 1.0].")
.createWithDefault(0.04)
+ val DYNAMIC_BLOOM_FILTER_JOIN_PRUNING_ENABLED =
+ buildConf("spark.sql.optimizer.dynamicBloomFilterJoinPruning.enabled")
+ .doc("When true, we will generate bloom filter predicate for column when it's used as join " +
Review comment:
nit: "generates a bloom filter predicate for a join key column"
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -353,6 +353,23 @@ object SQLConf {
.checkValue(ratio => ratio > 0.0 && ratio <= 1.0, "The ratio value must be in (0.0, 1.0].")
.createWithDefault(0.04)
+ val DYNAMIC_BLOOM_FILTER_JOIN_PRUNING_ENABLED =
+ buildConf("spark.sql.optimizer.dynamicBloomFilterJoinPruning.enabled")
+ .doc("When true, we will generate bloom filter predicate for column when it's used as join " +
+ "key. Note that, dynamic bloom filter join pruning only works if exchange reuse enabled.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val DYNAMIC_BLOOM_FILTER_JOIN_PRUNING_MAX_BLOOM_FILTER_ENTRIES =
+ buildConf("spark.sql.optimizer.dynamicBloomFilterJoinPruning.maxBloomFilterEntries")
+ .doc("The maximum number of bloom filter entries allowed when building dynamic bloom filter" +
Review comment:
nit: s/when building/for
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -353,6 +353,23 @@ object SQLConf {
.checkValue(ratio => ratio > 0.0 && ratio <= 1.0, "The ratio value must be in (0.0, 1.0].")
.createWithDefault(0.04)
+ val DYNAMIC_BLOOM_FILTER_JOIN_PRUNING_ENABLED =
+ buildConf("spark.sql.optimizer.dynamicBloomFilterJoinPruning.enabled")
+ .doc("When true, we will generate bloom filter predicate for column when it's used as join " +
+ "key. Note that, dynamic bloom filter join pruning only works if exchange reuse enabled.")
Review comment:
Use the config property so it becomes `s"only works with ${SQLConf.EXCHANGE_REUSE_ENABLED.key} enabled"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org