You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/12/01 15:33:56 UTC

[PR] Add additional test coverage of multi-value PartitionPruningStats [datafusion]

alamb opened a new pull request, #19021:
URL: https://github.com/apache/datafusion/pull/19021

   ## Which issue does this PR close?
   
   - Follow on to https://github.com/apache/datafusion/pull/18923
   
   ## Rationale for this change
   
   I was confused about some of the tests for `PartitionPruningStatistics` so let's add some
   more comments to explain what it is doing, and add additional coverage for multi-value columns
   
   
   ## What changes are included in this PR?
   
   Add a new test 
   
   ## Are these changes tested?
   
   Only tests 
   ## Are there any user-facing changes?
   
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add additional test coverage of multi-value PartitionPruningStats [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #19021:
URL: https://github.com/apache/datafusion/pull/19021#discussion_r2577598256


##########
datafusion/common/src/pruning.rs:
##########
@@ -562,26 +571,32 @@ mod tests {
 
     #[test]
     fn test_partition_pruning_statistics_multiple_positive_values() {
-        let partition_values = vec![
-            vec![ScalarValue::from(1i32), ScalarValue::from(2i32)],
-            vec![ScalarValue::from(3i32), ScalarValue::from(4i32)],
-        ];
-        let partition_fields = vec![
-            Arc::new(Field::new("a", DataType::Int32, false)),
-            Arc::new(Field::new("b", DataType::Int32, false)),
-        ];
-        let partition_stats =
-            PartitionPruningStatistics::try_new(partition_values, partition_fields)
-                .unwrap();
+        let partition_stats = partition_pruning_statistics_setup();
 
         let column_a = Column::new_unqualified("a");
 
+        // The two containers have `a` values 1 and 3, so they both only contain values from 1 and 3
         let values = HashSet::from([ScalarValue::from(1i32), ScalarValue::from(3i32)]);
         let contained_a = partition_stats.contained(&column_a, &values).unwrap();
         let expected_contained_a = BooleanArray::from(vec![true, true]);
         assert_eq!(contained_a, expected_contained_a);
     }
 
+    #[test]
+    fn test_partition_pruning_statistics_multiple_negative_values() {
+        let partition_stats = partition_pruning_statistics_setup();
+
+        let column_a = Column::new_unqualified("a");
+
+        // The two containers have `a` values 1 and 3,
+        // so the first contains ONLY values from 1,2
+        // but the second does not
+        let values = HashSet::from([ScalarValue::from(1i32), ScalarValue::from(2i32)]);

Review Comment:
   I was confused about  what should happens when the value of `a` isn't in the values too, so I added a test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add additional test coverage of multi-value PartitionPruningStats [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #19021:
URL: https://github.com/apache/datafusion/pull/19021


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org