You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "tmi (via GitHub)" <gi...@apache.org> on 2024/06/18 15:17:02 UTC

[PR] Minor: add parquet page stats for float{16, 32, 64} [datafusion]

tmi opened a new pull request, #10982:
URL: https://github.com/apache/datafusion/pull/10982

   ## Which issue does this PR close?
   
   Closes #10951.
   
   ## Rationale for this change
   
   Just extends the existing functionality for parquet page data stats -- `int` cases were covered previously, I just add `float` cases in the same fashion.
   
   ## What changes are included in this PR?
   
   * Float64 was a breeze, just add a macro invocation and case branch,
   * Float32 needed additionally an extension of data generation option -- I hope I didn't miss it existing already,
   * Float16 got a bit messy and required something I'm not happy with. Unlike other data types, it's physically represented as `FixedByteLength` of 2 bytes, which doesn't behave like a primitive type. In particular, I needed to add a `.clone()` at a particular place, which required going from `ident` to `expr` in a macro, affecting all cases. I'm very open to suggestions here -- I haven't come up with a better idea myself yet.
   
   ## Are these changes tested?
   
   Yes, I extended the existing test coverage to the newly added cases.
   
   ## Are there any user-facing changes?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add parquet page stats for float{16, 32, 64} [datafusion]

Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #10982:
URL: https://github.com/apache/datafusion/pull/10982#discussion_r1645033584


##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -614,6 +614,94 @@ async fn test_int_8() {
     .run();
 }
 
+#[tokio::test]
+async fn test_float_16() {
+    // This creates a parquet files of 1 column named f
+    let reader = TestReader {
+        scenario: Scenario::Float16,
+        row_per_group: 5,
+    }
+    .build()
+    .await;
+
+    Test {
+        reader: &reader,
+        // mins are [-5, -4, 0, 5]
+        expected_min: Arc::new(Float16Array::from(vec![
+            f16::from_f32(-5.),
+            f16::from_f32(-4.),
+            f16::from_f32(-0.),
+            f16::from_f32(5.),
+        ])),
+        // maxes are [-1, 0, 4, 9]
+        expected_max: Arc::new(Float16Array::from(vec![
+            f16::from_f32(-1.),
+            f16::from_f32(0.),
+            f16::from_f32(4.),
+            f16::from_f32(9.),
+        ])),
+        // nulls are [0, 0, 0, 0]
+        expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+        // row counts are [5, 5, 5, 5]
+        expected_row_counts: Some(UInt64Array::from(vec![5, 5, 5, 5])),
+        column_name: "f",
+        check: Check::Both,
+    }
+    .run();
+}
+
+#[tokio::test]
+async fn test_float_32() {
+    // This creates a parquet files of 1 column named f
+    let reader = TestReader {
+        scenario: Scenario::Float32,
+        row_per_group: 5,
+    }
+    .build()
+    .await;
+
+    Test {
+        reader: &reader,
+        // mins are [-5, -4, 0, 5]
+        expected_min: Arc::new(Float32Array::from(vec![-5., -4., -0., 5.0])),
+        // maxes are [-1, 0, 4, 9]
+        expected_max: Arc::new(Float32Array::from(vec![-1., 0., 4., 9.])),
+        // nulls are [0, 0, 0, 0]
+        expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+        // row counts are [5, 5, 5, 5]
+        expected_row_counts: Some(UInt64Array::from(vec![5, 5, 5, 5])),
+        column_name: "f",
+        check: Check::Both,
+    }
+    .run();
+}
+
+#[tokio::test]
+async fn test_float_64() {

Review Comment:
   There's a similar `test_float64` in this file that could similarly be updated or removed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add parquet page stats for float{16, 32, 64} [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10982:
URL: https://github.com/apache/datafusion/pull/10982#discussion_r1645026007


##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -614,6 +614,94 @@ async fn test_int_8() {
     .run();
 }
 
+#[tokio::test]

Review Comment:
   💯  very nice tests



##########
datafusion/core/src/datasource/physical_plan/parquet/statistics.rs:
##########
@@ -548,11 +549,66 @@ macro_rules! make_data_page_stats_iterator {
     };
 }
 
-make_data_page_stats_iterator!(MinInt32DataPageStatsIterator, min, Index::INT32, i32);
-make_data_page_stats_iterator!(MaxInt32DataPageStatsIterator, max, Index::INT32, i32);
-make_data_page_stats_iterator!(MinInt64DataPageStatsIterator, min, Index::INT64, i64);
-make_data_page_stats_iterator!(MaxInt64DataPageStatsIterator, max, Index::INT64, i64);
-
+make_data_page_stats_iterator!(
+    MinInt32DataPageStatsIterator,
+    |x: &PageIndex<i32>| { x.min },
+    Index::INT32,
+    i32
+);
+make_data_page_stats_iterator!(
+    MaxInt32DataPageStatsIterator,
+    |x: &PageIndex<i32>| { x.max },
+    Index::INT32,
+    i32
+);
+make_data_page_stats_iterator!(
+    MinInt64DataPageStatsIterator,
+    |x: &PageIndex<i64>| { x.min },
+    Index::INT64,
+    i64
+);
+make_data_page_stats_iterator!(
+    MaxInt64DataPageStatsIterator,
+    |x: &PageIndex<i64>| { x.max },
+    Index::INT64,
+    i64
+);
+make_data_page_stats_iterator!(
+    MinFloat16DataPageStatsIterator,
+    |x: &PageIndex<FixedLenByteArray>| { x.min.clone() },
+    Index::FIXED_LEN_BYTE_ARRAY,

Review Comment:
   cargo doc --document-private-items -p datafusion --open
   
   ![Screenshot 2024-06-18 at 4 32 10 PM](https://github.com/apache/datafusion/assets/490673/6a92c384-84e6-466e-b593-2e9c5313f7f6)
   
   I see what is going on -- I think this makes sense for now
   
   Implies that the iterator is an iterator over Vec<FixedLengthItem



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Add parquet page stats for float{16, 32, 64} [datafusion]

Posted by "tshauck (via GitHub)" <gi...@apache.org>.
tshauck commented on code in PR #10982:
URL: https://github.com/apache/datafusion/pull/10982#discussion_r1645032465


##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -614,6 +614,94 @@ async fn test_int_8() {
     .run();
 }
 
+#[tokio::test]

Review Comment:
   I think this test is mostly a duplicate of `test_float16` just that one has `Check::RowGroup` vs ``Check::Both`... maybe one or the other can be updated?



##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -614,6 +614,94 @@ async fn test_int_8() {
     .run();
 }
 
+#[tokio::test]

Review Comment:
   I think this test is mostly a duplicate of `test_float16` just that one has `Check::RowGroup` vs `Check::Both`... maybe one or the other can be updated?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Minor: add parquet page stats for float{16, 32, 64} [datafusion]

Posted by "tmi (via GitHub)" <gi...@apache.org>.
tmi commented on PR #10982:
URL: https://github.com/apache/datafusion/pull/10982#issuecomment-2176384402

   oh thx, haven't checked since yesterday...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Minor: add parquet page stats for float{16, 32, 64} [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10982:
URL: https://github.com/apache/datafusion/pull/10982#issuecomment-2176377128

   FYI there appears to be another PR open for this https://github.com/apache/datafusion/pull/10972


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org