You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "samuelcolvin (via GitHub)" <gi...@apache.org> on 2024/05/12 12:24:10 UTC

[PR] fix `null_count` on `compute_record_batch_statistics` [datafusion]

samuelcolvin opened a new pull request, #10468:
URL: https://github.com/apache/datafusion/pull/10468

   ## Rationale for this change
   
   Maybe I'm missing something or being dumb, but while reading `datafusion::physical_plan::common::compute_record_batch_statistics` I noticed that the `null_count` took the value from the last partition, not the sum of all partition.
   
   ## What changes are included in this PR?
   
   * fix `compute_record_batch_statistics`
   * add a test
   
   ## Are these changes tested?
   
   yes
   
   ## Are there any user-facing changes?
   
   I don't think so, perhaps some queries are now correct that were previously incorrect.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix `null_count` on `compute_record_batch_statistics` to report null counts across partitions [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #10468:
URL: https://github.com/apache/datafusion/pull/10468


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix `null_count` on `compute_record_batch_statistics` to report null counts across partitions [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10468:
URL: https://github.com/apache/datafusion/pull/10468#issuecomment-2109966353

   I took the liberty of merging up from main and runnning `cargo fmt` to fix the CI test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix `null_count` on `compute_record_batch_statistics` [datafusion]

Posted by "samuelcolvin (via GitHub)" <gi...@apache.org>.
samuelcolvin commented on code in PR #10468:
URL: https://github.com/apache/datafusion/pull/10468#discussion_r1597704992


##########
datafusion/physical-plan/src/common.rs:
##########
@@ -153,16 +153,23 @@ pub fn compute_record_batch_statistics(
         })
         .sum();
 
-    let mut column_statistics = vec![ColumnStatistics::new_unknown(); projection.len()];
+    let mut null_counts = vec![0; projection.len()];
 
     for partition in batches.iter() {
         for batch in partition {
             for (stat_index, col_index) in projection.iter().enumerate() {
-                column_statistics[stat_index].null_count =
-                    Precision::Exact(batch.column(*col_index).null_count());
+                null_counts[stat_index] += batch.column(*col_index).null_count();

Review Comment:
   There would be fewer bounds checks i think if we used `zip` here, rather than `.enumerate()` and lookup each column



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix `null_count` on `compute_record_batch_statistics` to report null counts across partitions [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10468:
URL: https://github.com/apache/datafusion/pull/10468#discussion_r1598897904


##########
datafusion/physical-plan/src/common.rs:
##########
@@ -687,4 +694,42 @@ mod tests {
         assert_eq!(actual, expected);
         Ok(())
     }
+
+    #[test]
+    fn test_compute_record_batch_statistics_null() -> Result<()> {
+        let schema = Arc::new(Schema::new(vec![
+            Field::new("u64", DataType::UInt64, true),
+        ]));
+        let batch1 = RecordBatch::try_new(
+            Arc::clone(&schema),
+            vec![
+                Arc::new(UInt64Array::from(vec![Some(1), None, None])),
+            ],
+        )?;
+        let batch2 = RecordBatch::try_new(
+            Arc::clone(&schema),
+            vec![
+                Arc::new(UInt64Array::from(vec![Some(1), Some(2), None])),
+            ],
+        )?;
+        let byte_size = batch1.get_array_memory_size() + batch2.get_array_memory_size();
+        let actual =
+            compute_record_batch_statistics(&[vec![batch1], vec![batch2]], &schema, None);
+
+        let expected = Statistics {
+            num_rows: Precision::Exact(6),
+            total_byte_size: Precision::Exact(byte_size),
+            column_statistics: vec![
+                ColumnStatistics {
+                    distinct_count: Precision::Absent,
+                    max_value: Precision::Absent,
+                    min_value: Precision::Absent,
+                    null_count: Precision::Exact(3),
+                },
+            ],
+        };
+
+        assert_eq!(actual, expected);

Review Comment:
   I verified test coverage by running this test without the code in this PR and the test fails like this
   
   ```
   
   assertion `left == right` failed
     left: Statistics { num_rows: Exact(6), total_byte_size: Exact(368), column_statistics: [ColumnStatistics { null_count: Exact(1), max_value: Absent, min_value: Absent, distinct_count: Absent }] }
    right: Statistics { num_rows: Exact(6), total_byte_size: Exact(368), column_statistics: [ColumnStatistics { null_count: Exact(3), max_value: Absent, min_value: Absent, distinct_count: Absent }] }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix `null_count` on `compute_record_batch_statistics` to report null counts across partitions [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10468:
URL: https://github.com/apache/datafusion/pull/10468#issuecomment-2116426288

   Thanks again @samuelcolvin  -- sorry for the delay in merging


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org