[GitHub] [spark] cloud-fan commented on a change in pull request #33650: [SPARK-36351][SQL] Refactor filter push down in file source v2

[GitBox <gi...@apache.org>]

cloud-fan commented on a change in pull request #33650:
URL: https://github.com/apache/spark/pull/33650#discussion_r732551001



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala
##########
@@ -40,37 +40,43 @@ object PushDownUtils extends PredicateHelper {
   def pushFilters(
       scanBuilder: ScanBuilder,
       filters: Seq[Expression]): (Seq[sources.Filter], Seq[Expression]) = {
+    // A map from translated data source leaf node filters to original catalyst filter
+    // expressions. For a `And`/`Or` predicate, it is possible that the predicate is partially
+    // pushed down. This map can be used to construct a catalyst filter expression from the
+    // input filter, or a superset(partial push down filter) of the input filter.

Review comment:
       Post-scan filters mean Spark should evaluate them. Pushed filters mean the data source should handle them. These two can overlap. For example, data filters are evaluated by both parquet reader and Spark (because parquet does row group filtering, not per-row).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org