[GitHub] [spark] wangyum commented on a change in pull request #31984: [SPARK-34884][SQL] Improve dynamic partition pruning evaluation

[GitBox <gi...@apache.org>]

wangyum commented on a change in pull request #31984:
URL: https://github.com/apache/spark/pull/31984#discussion_r603167405



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PartitionPruning.scala
##########
@@ -48,6 +49,10 @@ import org.apache.spark.sql.execution.datasources.{HadoopFsRelation, LogicalRela
  */
 object PartitionPruning extends Rule[LogicalPlan] with PredicateHelper with JoinSelectionHelper {
 
+  private val buildBroadcastThreshold = math.max(
+    AUTO_BROADCASTJOIN_THRESHOLD.defaultValue.getOrElse(conf.autoBroadcastJoinThreshold),

Review comment:
       To avoid disable DPP by setting `autoBroadcastJoinThreshold ` to a small value. For example:
   1. If user set autoBroadcastJoinThreshold to -1 and has benefit:
      Before this PR:
      spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly | filtering side data size <= 10MB | filtering side data size > 10MB
      -- | -- | --
      true | N | N
      false | Y | Y
   
      After this PR:
      spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly | filtering side data size <= 10MB | filtering side data size > 10MB
      -- | -- | --
      true | N | N
      false | Y | N
   
   
   2. If user set autoBroadcastJoinThreshold to 100MB and has benefit:
      Before this PR:
      spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly | filtering side data size <= 100MB | filtering side data size > 100MB
      -- | -- | --
      true | Y | N
      false | Y | Y
   
      After this PR:
      spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly | filtering side data size <= 100MB | filtering side data size > 100MB
      -- | -- | --
      true | Y | N
      false | Y | N
   
   `Y` means will insert a DPP, `N` means will not insert a DPP. It is much reasonable than before. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org