[GitHub] [arrow-datafusion] NGA-TRAN commented on a change in pull request #426: Rewrite pruning logic in terms of PruningStatistics using Array trait (option 2)

[GitBox <gi...@apache.org>]

NGA-TRAN commented on a change in pull request #426:
URL: https://github.com/apache/arrow-datafusion/pull/426#discussion_r638935367



##########
File path: datafusion/src/physical_optimizer/pruning.rs
##########
@@ -52,26 +44,81 @@ use crate::{
     physical_plan::{planner::DefaultPhysicalPlanner, ColumnarValue, PhysicalExpr},
 };
 
+/// Interface to pass statistics information to [`PruningPredicates`]
+///
+/// Returns statistics for containers / files of data in Arrays.
+///
+/// For example, for the following three files with a single column
+/// ```text
+/// file1: column a: min=5, max=10
+/// file2: column a: No stats
+/// file2: column a: min=20, max=30
+/// ```
+///
+/// PruningStatistics should return:
+///
+/// ```text
+/// min_values("a") -> Some([5, Null, 20])
+/// max_values("a") -> Some([20, Null, 30])
+/// min_values("X") -> None

Review comment:
       What is line for?

##########
File path: datafusion/src/physical_optimizer/pruning.rs
##########
@@ -52,26 +44,81 @@ use crate::{
     physical_plan::{planner::DefaultPhysicalPlanner, ColumnarValue, PhysicalExpr},
 };
 
+/// Interface to pass statistics information to [`PruningPredicates`]
+///
+/// Returns statistics for containers / files of data in Arrays.
+///
+/// For example, for the following three files with a single column
+/// ```text
+/// file1: column a: min=5, max=10
+/// file2: column a: No stats
+/// file2: column a: min=20, max=30
+/// ```
+///
+/// PruningStatistics should return:
+///
+/// ```text
+/// min_values("a") -> Some([5, Null, 20])
+/// max_values("a") -> Some([20, Null, 30])

Review comment:
       ```suggestion
   /// max_values("a") -> Some([10, Null, 30])
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org