[GitHub] [spark] huaxingao commented on a change in pull request #34062: [SPARK-36819][SQL] Don't insert redundant filters in case static partition pruning can be done

[GitBox <gi...@apache.org>]

huaxingao commented on a change in pull request #34062:
URL: https://github.com/apache/spark/pull/34062#discussion_r714422174



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PartitionPruning.scala
##########
@@ -201,26 +201,46 @@ object PartitionPruning extends Rule[LogicalPlan] with PredicateHelper with Join
   }
 
   /**
-   * Returns whether an expression is likely to be selective
+   * Returns whether an expression is likely to be selective in dynamic partition filtering.
+   * 1. the predicate is selective.
+   * 2. the filtering predicate must not be subset of join key. In case it is, key then partition
+   * filter. can be inferred statically in optimization phase, hence return false.

Review comment:
       Sorry, I got confused here. Do you mean `In case it is, then partition filter can be inferred ...`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org