[GitHub] [arrow-datafusion] alamb commented on a change in pull request #689: Remove qualifiers on pushed down predicates / Fix parquet pruning

[GitBox <gi...@apache.org>]

alamb commented on a change in pull request #689:
URL: https://github.com/apache/arrow-datafusion/pull/689#discussion_r664718765



##########
File path: datafusion/src/physical_plan/planner.rs
##########
@@ -312,7 +313,13 @@ impl DefaultPhysicalPlanner {
                 filters,
                 limit,
                 ..
-            } => source.scan(projection, batch_size, filters, *limit),
+            } => {
+                // Remove all qualifiers from the scan as the provider
+                // doesn't know (nor should care) how the relation was
+                // referred to in the query
+                let filters = unnormalize_cols(filters.iter().cloned());

Review comment:
       Here is the actual fix for parquet pruning. 

##########
File path: datafusion/src/logical_plan/expr.rs
##########
@@ -1120,7 +1120,7 @@ pub fn columnize_expr(e: Expr, input_schema: &DFSchema) -> Expr {
 
 /// Recursively call [`Column::normalize`] on all Column expressions
 /// in the `expr` expression tree.
-pub fn normalize_col(e: Expr, schemas: &[&DFSchemaRef]) -> Result<Expr> {
+pub fn normalize_col(expr: Expr, schemas: &[&DFSchemaRef]) -> Result<Expr> {

Review comment:
       Changed to use the name `expr` to be more consistent with the rest of the codebase

##########
File path: datafusion/src/logical_plan/expr.rs
##########
@@ -1762,4 +1794,76 @@ mod tests {
             }
         }
     }
+
+    #[test]
+    fn normalize_cols() {

Review comment:
       I added some additional unit test coverage here for normalize when I was writing the `unnormalize` versions




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org