You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "NGA-TRAN (via GitHub)" <gi...@apache.org> on 2024/05/15 21:41:31 UTC
[PR] test: some tests to write data to a parquet file and read its metadata [datafusion]
NGA-TRAN opened a new pull request, #10537:
URL: https://github.com/apache/datafusion/pull/10537
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
First PR of #https://github.com/apache/datafusion/issues/10453
Note: this is just a draft for @alamb to confirm this is what he expects
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
Tests that write data into a parquet file then reads and verify its metadata
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
Yes
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
Not yet
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: API for collecting statistics/index for metadata of a parquet file + tests [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1607110465
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,652 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::fs::File;
+use std::sync::Arc;
+
+use arrow_array::{
+ make_array, Array, ArrayRef, Int16Array, Int32Array, Int64Array, Int8Array,
+ RecordBatch, UInt64Array,
+};
+use arrow_schema::{DataType, Field, Schema};
+use datafusion::datasource::physical_plan::parquet::{
+ RequestedStatistics, StatisticsConverter,
+};
+use parquet::arrow::arrow_reader::{ArrowReaderBuilder, ParquetRecordBatchReaderBuilder};
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+
+use crate::parquet::Scenario;
+
+use super::make_test_file_rg;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_file_one_column(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![make_int64_batches_with_null(
+ num_null,
+ no_null_values_start,
+ no_null_values_end,
+ )];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ // close file
+ let _file_meta = writer.close().unwrap();
+
+ // open the file & get the reader
+ let file = output_file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+// Create a parquet file with many columns each has different data type
+// - Data types are specified by the given scenario
+// - Row group sizes are withe the same or different depending on the provided row_per_group & data created in the scenario
+pub async fn parquet_file_many_columns(
+ scenario: super::Scenario,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let file = make_test_file_rg(scenario, row_per_group).await;
+
+ // open the file & get the reader
+ let file = file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+struct Test {
+ reader: ParquetRecordBatchReaderBuilder<File>,
+ expected_min: ArrayRef,
+ expected_max: ArrayRef,
+ expected_null_counts: UInt64Array,
+ expected_row_counts: UInt64Array,
+}
+
+impl Test {
+ fn run(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min,
+ expected_max,
+ expected_null_counts,
+ expected_row_counts,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+
+ assert_eq!(&min, &expected_min, "Mismatch with expected minimums");
+
+ let max = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Max,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ assert_eq!(&max, &expected_max, "Mismatch with expected maximum");
+
+ let null_counts = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::NullCount,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ let expected_null_counts = Arc::new(expected_null_counts) as ArrayRef;
+ assert_eq!(
+ &null_counts, &expected_null_counts,
+ "Mismatch with expected null counts"
+ );
+
+ let row_counts = StatisticsConverter::row_counts(reader.metadata()).unwrap();
+ assert_eq!(
+ row_counts, expected_row_counts,
+ "Mismatch with expected row counts"
+ );
+ }
+
+ fn run_col_not_found(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min: _,
+ expected_max: _,
+ expected_null_counts: _,
+ expected_row_counts: _,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ );
+
+ assert!(min.is_err());
+ }
+}
+
+// TESTS
+//
+// Remaining cases
+// - Create parquet files / metadata with missing statistic values
+// - Create parquet files / metadata with different data types -- included but not all data types yet
+// - Create parquet files / metadata with different row group sizes -- done
+// - Using truncated statistics ("exact min value" and "exact max value" https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.max_is_exact)
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(0, 4, 7, row_per_group);
+ Test {
+ reader,
+ // min is 4
+ expected_min: Arc::new(Int64Array::from(vec![4])),
+ // max is 6
+ expected_max: Arc::new(Int64Array::from(vec![6])),
+ // no nulls
+ expected_null_counts: UInt64Array::from(vec![0]),
+ // 3 rows
+ expected_row_counts: UInt64Array::from(vec![3]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_one_row_group_with_null_and_negative() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(2, -1, 5, row_per_group);
+
+ Test {
+ reader,
+ // min is -1
+ expected_min: Arc::new(Int64Array::from(vec![-1])),
+ // max is 4
+ expected_max: Arc::new(Int64Array::from(vec![4])),
+ // 2 nulls
+ expected_null_counts: UInt64Array::from(vec![2]),
+ // 8 rows
+ expected_row_counts: UInt64Array::from(vec![8]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_group_with_null() {
+ let row_per_group = 10;
+ let reader = parquet_file_one_column(2, 4, 17, row_per_group);
+
+ Test {
+ reader,
+ // mins are [4, 14]
+ expected_min: Arc::new(Int64Array::from(vec![4, 14])),
+ // maxes are [13, 16]
+ expected_max: Arc::new(Int64Array::from(vec![13, 16])),
+ // nulls are [0, 2]
+ expected_null_counts: UInt64Array::from(vec![0, 2]),
+ // row counts are [10, 5]
+ expected_row_counts: UInt64Array::from(vec![10, 5]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_groups_with_all_nulls_in_one() {
+ let row_per_group = 5;
+ let reader = parquet_file_one_column(4, -2, 2, row_per_group);
+
+ Test {
+ reader,
+ // mins are [-2, null]
+ expected_min: Arc::new(Int64Array::from(vec![Some(-2), None])),
+ // maxes are [1, null]
+ expected_max: Arc::new(Int64Array::from(vec![Some(1), None])),
+ // nulls are [1, 3]
+ expected_null_counts: UInt64Array::from(vec![1, 3]),
+ // row counts are [5, 3]
+ expected_row_counts: UInt64Array::from(vec![5, 3]),
+ }
+ .run("i64")
+}
+
+/////////////// MORE GENERAL TESTS //////////////////////
+// . Many columns in a file
+// . Differnet data types
+// . Different row group sizes
+
+// Four different integer types
+#[tokio::test]
+async fn test_int_64() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i64");
+}
+
+#[tokio::test]
+async fn test_int_32() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int32Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int32Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i32");
+}
+
+// BUG: ignore this test for now
+// Note that the file has 4 columns named "i8", "i16", "i32", "i64".
+// - The tests on column i32 and i64 passed.
+// - The tests on column i8 and i16 failed.
+#[ignore]
+#[tokio::test]
+async fn test_int_16() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
Review Comment:
Filed https://github.com/apache/datafusion/issues/10585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: API for collecting statistics/index for metadata of a parquet file + tests [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1607107595
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,652 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::fs::File;
+use std::sync::Arc;
+
+use arrow_array::{
+ make_array, Array, ArrayRef, Int16Array, Int32Array, Int64Array, Int8Array,
+ RecordBatch, UInt64Array,
+};
+use arrow_schema::{DataType, Field, Schema};
+use datafusion::datasource::physical_plan::parquet::{
+ RequestedStatistics, StatisticsConverter,
+};
+use parquet::arrow::arrow_reader::{ArrowReaderBuilder, ParquetRecordBatchReaderBuilder};
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+
+use crate::parquet::Scenario;
+
+use super::make_test_file_rg;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_file_one_column(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![make_int64_batches_with_null(
+ num_null,
+ no_null_values_start,
+ no_null_values_end,
+ )];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ // close file
+ let _file_meta = writer.close().unwrap();
+
+ // open the file & get the reader
+ let file = output_file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+// Create a parquet file with many columns each has different data type
+// - Data types are specified by the given scenario
+// - Row group sizes are withe the same or different depending on the provided row_per_group & data created in the scenario
+pub async fn parquet_file_many_columns(
+ scenario: super::Scenario,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let file = make_test_file_rg(scenario, row_per_group).await;
+
+ // open the file & get the reader
+ let file = file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+struct Test {
+ reader: ParquetRecordBatchReaderBuilder<File>,
+ expected_min: ArrayRef,
+ expected_max: ArrayRef,
+ expected_null_counts: UInt64Array,
+ expected_row_counts: UInt64Array,
+}
+
+impl Test {
+ fn run(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min,
+ expected_max,
+ expected_null_counts,
+ expected_row_counts,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+
+ assert_eq!(&min, &expected_min, "Mismatch with expected minimums");
+
+ let max = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Max,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ assert_eq!(&max, &expected_max, "Mismatch with expected maximum");
+
+ let null_counts = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::NullCount,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ let expected_null_counts = Arc::new(expected_null_counts) as ArrayRef;
+ assert_eq!(
+ &null_counts, &expected_null_counts,
+ "Mismatch with expected null counts"
+ );
+
+ let row_counts = StatisticsConverter::row_counts(reader.metadata()).unwrap();
+ assert_eq!(
+ row_counts, expected_row_counts,
+ "Mismatch with expected row counts"
+ );
+ }
+
+ fn run_col_not_found(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min: _,
+ expected_max: _,
+ expected_null_counts: _,
+ expected_row_counts: _,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ );
+
+ assert!(min.is_err());
+ }
+}
+
+// TESTS
+//
+// Remaining cases
+// - Create parquet files / metadata with missing statistic values
+// - Create parquet files / metadata with different data types -- included but not all data types yet
+// - Create parquet files / metadata with different row group sizes -- done
+// - Using truncated statistics ("exact min value" and "exact max value" https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.max_is_exact)
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(0, 4, 7, row_per_group);
+ Test {
+ reader,
+ // min is 4
+ expected_min: Arc::new(Int64Array::from(vec![4])),
+ // max is 6
+ expected_max: Arc::new(Int64Array::from(vec![6])),
+ // no nulls
+ expected_null_counts: UInt64Array::from(vec![0]),
+ // 3 rows
+ expected_row_counts: UInt64Array::from(vec![3]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_one_row_group_with_null_and_negative() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(2, -1, 5, row_per_group);
+
+ Test {
+ reader,
+ // min is -1
+ expected_min: Arc::new(Int64Array::from(vec![-1])),
+ // max is 4
+ expected_max: Arc::new(Int64Array::from(vec![4])),
+ // 2 nulls
+ expected_null_counts: UInt64Array::from(vec![2]),
+ // 8 rows
+ expected_row_counts: UInt64Array::from(vec![8]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_group_with_null() {
+ let row_per_group = 10;
+ let reader = parquet_file_one_column(2, 4, 17, row_per_group);
+
+ Test {
+ reader,
+ // mins are [4, 14]
+ expected_min: Arc::new(Int64Array::from(vec![4, 14])),
+ // maxes are [13, 16]
+ expected_max: Arc::new(Int64Array::from(vec![13, 16])),
+ // nulls are [0, 2]
+ expected_null_counts: UInt64Array::from(vec![0, 2]),
+ // row counts are [10, 5]
+ expected_row_counts: UInt64Array::from(vec![10, 5]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_groups_with_all_nulls_in_one() {
+ let row_per_group = 5;
+ let reader = parquet_file_one_column(4, -2, 2, row_per_group);
+
+ Test {
+ reader,
+ // mins are [-2, null]
+ expected_min: Arc::new(Int64Array::from(vec![Some(-2), None])),
+ // maxes are [1, null]
+ expected_max: Arc::new(Int64Array::from(vec![Some(1), None])),
+ // nulls are [1, 3]
+ expected_null_counts: UInt64Array::from(vec![1, 3]),
+ // row counts are [5, 3]
+ expected_row_counts: UInt64Array::from(vec![5, 3]),
+ }
+ .run("i64")
+}
+
+/////////////// MORE GENERAL TESTS //////////////////////
+// . Many columns in a file
+// . Differnet data types
+// . Different row group sizes
+
+// Four different integer types
+#[tokio::test]
+async fn test_int_64() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i64");
+}
+
+#[tokio::test]
+async fn test_int_32() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int32Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int32Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i32");
+}
+
+// BUG: ignore this test for now
+// Note that the file has 4 columns named "i8", "i16", "i32", "i64".
Review Comment:
```suggestion
// https://github.com/apache/datafusion/issues/10585
// Note that the file has 4 columns named "i8", "i16", "i32", "i64".
```
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,652 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::fs::File;
+use std::sync::Arc;
+
+use arrow_array::{
+ make_array, Array, ArrayRef, Int16Array, Int32Array, Int64Array, Int8Array,
+ RecordBatch, UInt64Array,
+};
+use arrow_schema::{DataType, Field, Schema};
+use datafusion::datasource::physical_plan::parquet::{
+ RequestedStatistics, StatisticsConverter,
+};
+use parquet::arrow::arrow_reader::{ArrowReaderBuilder, ParquetRecordBatchReaderBuilder};
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+
+use crate::parquet::Scenario;
+
+use super::make_test_file_rg;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_file_one_column(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![make_int64_batches_with_null(
+ num_null,
+ no_null_values_start,
+ no_null_values_end,
+ )];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ // close file
+ let _file_meta = writer.close().unwrap();
+
+ // open the file & get the reader
+ let file = output_file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+// Create a parquet file with many columns each has different data type
+// - Data types are specified by the given scenario
+// - Row group sizes are withe the same or different depending on the provided row_per_group & data created in the scenario
+pub async fn parquet_file_many_columns(
+ scenario: super::Scenario,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let file = make_test_file_rg(scenario, row_per_group).await;
+
+ // open the file & get the reader
+ let file = file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+struct Test {
+ reader: ParquetRecordBatchReaderBuilder<File>,
+ expected_min: ArrayRef,
+ expected_max: ArrayRef,
+ expected_null_counts: UInt64Array,
+ expected_row_counts: UInt64Array,
+}
+
+impl Test {
+ fn run(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min,
+ expected_max,
+ expected_null_counts,
+ expected_row_counts,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+
+ assert_eq!(&min, &expected_min, "Mismatch with expected minimums");
+
+ let max = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Max,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ assert_eq!(&max, &expected_max, "Mismatch with expected maximum");
+
+ let null_counts = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::NullCount,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ let expected_null_counts = Arc::new(expected_null_counts) as ArrayRef;
+ assert_eq!(
+ &null_counts, &expected_null_counts,
+ "Mismatch with expected null counts"
+ );
+
+ let row_counts = StatisticsConverter::row_counts(reader.metadata()).unwrap();
+ assert_eq!(
+ row_counts, expected_row_counts,
+ "Mismatch with expected row counts"
+ );
+ }
+
+ fn run_col_not_found(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min: _,
+ expected_max: _,
+ expected_null_counts: _,
+ expected_row_counts: _,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ );
+
+ assert!(min.is_err());
+ }
+}
+
+// TESTS
+//
+// Remaining cases
+// - Create parquet files / metadata with missing statistic values
+// - Create parquet files / metadata with different data types -- included but not all data types yet
+// - Create parquet files / metadata with different row group sizes -- done
+// - Using truncated statistics ("exact min value" and "exact max value" https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.max_is_exact)
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(0, 4, 7, row_per_group);
+ Test {
+ reader,
+ // min is 4
+ expected_min: Arc::new(Int64Array::from(vec![4])),
+ // max is 6
+ expected_max: Arc::new(Int64Array::from(vec![6])),
+ // no nulls
+ expected_null_counts: UInt64Array::from(vec![0]),
+ // 3 rows
+ expected_row_counts: UInt64Array::from(vec![3]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_one_row_group_with_null_and_negative() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(2, -1, 5, row_per_group);
+
+ Test {
+ reader,
+ // min is -1
+ expected_min: Arc::new(Int64Array::from(vec![-1])),
+ // max is 4
+ expected_max: Arc::new(Int64Array::from(vec![4])),
+ // 2 nulls
+ expected_null_counts: UInt64Array::from(vec![2]),
+ // 8 rows
+ expected_row_counts: UInt64Array::from(vec![8]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_group_with_null() {
+ let row_per_group = 10;
+ let reader = parquet_file_one_column(2, 4, 17, row_per_group);
+
+ Test {
+ reader,
+ // mins are [4, 14]
+ expected_min: Arc::new(Int64Array::from(vec![4, 14])),
+ // maxes are [13, 16]
+ expected_max: Arc::new(Int64Array::from(vec![13, 16])),
+ // nulls are [0, 2]
+ expected_null_counts: UInt64Array::from(vec![0, 2]),
+ // row counts are [10, 5]
+ expected_row_counts: UInt64Array::from(vec![10, 5]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_groups_with_all_nulls_in_one() {
+ let row_per_group = 5;
+ let reader = parquet_file_one_column(4, -2, 2, row_per_group);
+
+ Test {
+ reader,
+ // mins are [-2, null]
+ expected_min: Arc::new(Int64Array::from(vec![Some(-2), None])),
+ // maxes are [1, null]
+ expected_max: Arc::new(Int64Array::from(vec![Some(1), None])),
+ // nulls are [1, 3]
+ expected_null_counts: UInt64Array::from(vec![1, 3]),
+ // row counts are [5, 3]
+ expected_row_counts: UInt64Array::from(vec![5, 3]),
+ }
+ .run("i64")
+}
+
+/////////////// MORE GENERAL TESTS //////////////////////
+// . Many columns in a file
+// . Differnet data types
+// . Different row group sizes
+
+// Four different integer types
+#[tokio::test]
+async fn test_int_64() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i64");
+}
+
+#[tokio::test]
+async fn test_int_32() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int32Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int32Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i32");
+}
+
+// BUG: ignore this test for now
+// Note that the file has 4 columns named "i8", "i16", "i32", "i64".
+// - The tests on column i32 and i64 passed.
+// - The tests on column i8 and i16 failed.
+#[ignore]
+#[tokio::test]
+async fn test_int_16() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array type even though I debugged and the columns name is "i16" an its data is Int16
+ // My debugging tells me the bug is either at:
+ // 1. The new code to get "iter". See the code in this PR with
+ // // Get an iterator over the column statistics
+ // let iter = row_groups
+ // .iter()
+ // .map(|x| x.column(parquet_idx).statistics());
+ // OR
+ // 2. in the function (and/or its marco) `pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics>>>` here
+ // https://github.com/apache/datafusion/blob/ea023e2d4878240eece870cf4b346c7a0667aeed/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs#L179
+ expected_min: Arc::new(Int16Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int16Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i16");
+}
+
+// BUG (same as above): ignore this test for now
+#[ignore]
Review Comment:
```suggestion
// https://github.com/apache/datafusion/issues/10585
#[ignore]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: API for collecting statistics/index for metadata of a parquet file + tests [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1607104991
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,652 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::fs::File;
+use std::sync::Arc;
+
+use arrow_array::{
+ make_array, Array, ArrayRef, Int16Array, Int32Array, Int64Array, Int8Array,
+ RecordBatch, UInt64Array,
+};
+use arrow_schema::{DataType, Field, Schema};
+use datafusion::datasource::physical_plan::parquet::{
+ RequestedStatistics, StatisticsConverter,
+};
+use parquet::arrow::arrow_reader::{ArrowReaderBuilder, ParquetRecordBatchReaderBuilder};
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+
+use crate::parquet::Scenario;
+
+use super::make_test_file_rg;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_file_one_column(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![make_int64_batches_with_null(
+ num_null,
+ no_null_values_start,
+ no_null_values_end,
+ )];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ // close file
+ let _file_meta = writer.close().unwrap();
+
+ // open the file & get the reader
+ let file = output_file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+// Create a parquet file with many columns each has different data type
+// - Data types are specified by the given scenario
+// - Row group sizes are withe the same or different depending on the provided row_per_group & data created in the scenario
+pub async fn parquet_file_many_columns(
+ scenario: super::Scenario,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let file = make_test_file_rg(scenario, row_per_group).await;
+
+ // open the file & get the reader
+ let file = file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+struct Test {
+ reader: ParquetRecordBatchReaderBuilder<File>,
+ expected_min: ArrayRef,
+ expected_max: ArrayRef,
+ expected_null_counts: UInt64Array,
+ expected_row_counts: UInt64Array,
+}
+
+impl Test {
+ fn run(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min,
+ expected_max,
+ expected_null_counts,
+ expected_row_counts,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+
+ assert_eq!(&min, &expected_min, "Mismatch with expected minimums");
+
+ let max = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Max,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ assert_eq!(&max, &expected_max, "Mismatch with expected maximum");
+
+ let null_counts = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::NullCount,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ let expected_null_counts = Arc::new(expected_null_counts) as ArrayRef;
+ assert_eq!(
+ &null_counts, &expected_null_counts,
+ "Mismatch with expected null counts"
+ );
+
+ let row_counts = StatisticsConverter::row_counts(reader.metadata()).unwrap();
+ assert_eq!(
+ row_counts, expected_row_counts,
+ "Mismatch with expected row counts"
+ );
+ }
+
+ fn run_col_not_found(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min: _,
+ expected_max: _,
+ expected_null_counts: _,
+ expected_row_counts: _,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ );
+
+ assert!(min.is_err());
+ }
+}
+
+// TESTS
+//
+// Remaining cases
+// - Create parquet files / metadata with missing statistic values
+// - Create parquet files / metadata with different data types -- included but not all data types yet
+// - Create parquet files / metadata with different row group sizes -- done
+// - Using truncated statistics ("exact min value" and "exact max value" https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.max_is_exact)
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(0, 4, 7, row_per_group);
+ Test {
+ reader,
+ // min is 4
+ expected_min: Arc::new(Int64Array::from(vec![4])),
+ // max is 6
+ expected_max: Arc::new(Int64Array::from(vec![6])),
+ // no nulls
+ expected_null_counts: UInt64Array::from(vec![0]),
+ // 3 rows
+ expected_row_counts: UInt64Array::from(vec![3]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_one_row_group_with_null_and_negative() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(2, -1, 5, row_per_group);
+
+ Test {
+ reader,
+ // min is -1
+ expected_min: Arc::new(Int64Array::from(vec![-1])),
+ // max is 4
+ expected_max: Arc::new(Int64Array::from(vec![4])),
+ // 2 nulls
+ expected_null_counts: UInt64Array::from(vec![2]),
+ // 8 rows
+ expected_row_counts: UInt64Array::from(vec![8]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_group_with_null() {
+ let row_per_group = 10;
+ let reader = parquet_file_one_column(2, 4, 17, row_per_group);
+
+ Test {
+ reader,
+ // mins are [4, 14]
+ expected_min: Arc::new(Int64Array::from(vec![4, 14])),
+ // maxes are [13, 16]
+ expected_max: Arc::new(Int64Array::from(vec![13, 16])),
+ // nulls are [0, 2]
+ expected_null_counts: UInt64Array::from(vec![0, 2]),
+ // row counts are [10, 5]
+ expected_row_counts: UInt64Array::from(vec![10, 5]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_groups_with_all_nulls_in_one() {
+ let row_per_group = 5;
+ let reader = parquet_file_one_column(4, -2, 2, row_per_group);
+
+ Test {
+ reader,
+ // mins are [-2, null]
+ expected_min: Arc::new(Int64Array::from(vec![Some(-2), None])),
+ // maxes are [1, null]
+ expected_max: Arc::new(Int64Array::from(vec![Some(1), None])),
+ // nulls are [1, 3]
+ expected_null_counts: UInt64Array::from(vec![1, 3]),
+ // row counts are [5, 3]
+ expected_row_counts: UInt64Array::from(vec![5, 3]),
+ }
+ .run("i64")
+}
+
+/////////////// MORE GENERAL TESTS //////////////////////
+// . Many columns in a file
+// . Differnet data types
+// . Different row group sizes
+
+// Four different integer types
+#[tokio::test]
+async fn test_int_64() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i64");
+}
+
+#[tokio::test]
+async fn test_int_32() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int32Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int32Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i32");
+}
+
+// BUG: ignore this test for now
+// Note that the file has 4 columns named "i8", "i16", "i32", "i64".
+// - The tests on column i32 and i64 passed.
+// - The tests on column i8 and i16 failed.
+#[ignore]
+#[tokio::test]
+async fn test_int_16() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
Review Comment:
This is a good hint for me to understand the issue more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: some tests to write data to a parquet file and read its metadata [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1604129351
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,270 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::fs::File;
+use std::sync::Arc;
+
+use arrow_array::{make_array, Array, ArrayRef, Int64Array, RecordBatch, UInt64Array};
+use arrow_schema::{DataType, Field, Schema};
+use datafusion::datasource::physical_plan::parquet::arrow_statistics::{
+ self, parquet_stats_to_arrow,
+};
+use parquet::arrow::arrow_reader::{ArrowReaderBuilder, ParquetRecordBatchReaderBuilder};
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_file(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![
+ make_int64_batches_with_null(num_null, no_null_values_start, no_null_values_end), // TODO: likely make this more general using
+ // create_data_batch(scenario); where scenario is the respective enum data type
+ ];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ // close file
+ let _file_meta = writer.close().unwrap();
+
+ // open the file & get the reader
+ let file = output_file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+// TESTS
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let reader = parquet_file(0, 4, 7, row_per_group);
+
+ let statistics =
+ arrow_statistics::ParquetColumnStatistics::from_parquet_statistics(&reader)
+ .expect("getting statistics");
+
+ // only 1 column
+ assert_eq!(statistics.len(), 1, "expected 1 column");
+
+ let arrow_stats = parquet_stats_to_arrow(&DataType::Int64, &statistics[0]).unwrap();
+ println!("Arrow statistics:\n {arrow_stats:#?}");
+
+ // only one row group
+ let min = arrow_stats.min();
+ assert_eq!(arrow_stats.min().len(), 1, "expected 1 row group");
+
+ // min is 4
+ let expect = Int64Array::from(vec![4]);
+ let expect = Arc::new(expect) as ArrayRef;
+ assert_eq!(min, &expect);
+
+ // max is 6
+ let expect = Int64Array::from(vec![6]);
+ let expect = Arc::new(expect) as ArrayRef;
+ assert_eq!(arrow_stats.max(), &expect);
+
+ // no nulls
+ let expect = UInt64Array::from(vec![0]);
+ let expect = Arc::new(expect) as ArrayRef;
+ assert_eq!(arrow_stats.null_count(), &expect);
+
+ // 3 rows
+ let expect = UInt64Array::from(vec![3]);
+ let expect = Arc::new(expect) as ArrayRef;
+ assert_eq!(arrow_stats.row_count(), &expect);
+}
+
+#[tokio::test]
+async fn test_one_row_group_with_null_and_negative() {
+ let row_per_group = 20;
+ let reader = parquet_file(2, -1, 5, row_per_group);
+
+ let statistics =
+ arrow_statistics::ParquetColumnStatistics::from_parquet_statistics(&reader)
+ .expect("getting statistics");
+
+ // only 1 column
+ assert_eq!(statistics.len(), 1, "expected 1 column");
+
+ let arrow_stats = parquet_stats_to_arrow(&DataType::Int64, &statistics[0]).unwrap();
+ println!("Arrow statistics:\n {arrow_stats:#?}");
+
+ // only one row group
+ let min = arrow_stats.min();
+ assert_eq!(arrow_stats.min().len(), 1, "expected 1 row group");
+
+ // min is -1
+ let expect = Int64Array::from(vec![-1]);
+ let expect = Arc::new(expect) as ArrayRef;
+ assert_eq!(min, &expect);
+
+ // max is 4
+ let expect = Int64Array::from(vec![4]);
+ let expect = Arc::new(expect) as ArrayRef;
+ assert_eq!(arrow_stats.max(), &expect);
+
+ // 2 nulls
+ let expect = UInt64Array::from(vec![2]);
+ let expect = Arc::new(expect) as ArrayRef;
+ assert_eq!(arrow_stats.null_count(), &expect);
+
+ // 8 rows
+ let expect = UInt64Array::from(vec![8]);
+ let expect = Arc::new(expect) as ArrayRef;
+ assert_eq!(arrow_stats.row_count(), &expect);
+}
+
+#[tokio::test]
+async fn test_two_row_group_with_null() {
+ let row_per_group = 10;
+ let reader = parquet_file(2, 4, 17, row_per_group);
+
+ let statistics =
+ arrow_statistics::ParquetColumnStatistics::from_parquet_statistics(&reader)
+ .expect("getting statistics");
+
+ // only 1 column
+ assert_eq!(statistics.len(), 1, "expected 1 column");
+
+ let arrow_stats = parquet_stats_to_arrow(&DataType::Int64, &statistics[0]).unwrap();
+ println!("Arrow statistics:\n {arrow_stats:#?}");
+
+ // 2 row groups
+ let mins = arrow_stats.min();
+ assert_eq!(mins.len(), 2, "expected 2 row groups");
+
+ // mins are [4, 14]
+ let expect = Int64Array::from(vec![4, 14]);
+ let expect = Arc::new(expect) as ArrayRef;
+ assert_eq!(mins, &expect);
+
+ // maxs are [13, 16]
+ let maxs = arrow_stats.max();
+ let expect = Int64Array::from(vec![13, 16]);
+ let expect = Arc::new(expect) as ArrayRef;
+ assert_eq!(maxs, &expect);
+
+ // nuls are [0, 2]
+ let nulls = arrow_stats.null_count();
+ let expect = UInt64Array::from(vec![0, 2]);
+ let expect = Arc::new(expect) as ArrayRef;
+ assert_eq!(nulls, &expect);
+
+ // row counts are [10, 5]
+ let row_counts = arrow_stats.row_count();
+ let expect = UInt64Array::from(vec![10, 5]);
+ let expect = Arc::new(expect) as ArrayRef;
+ assert_eq!(row_counts, &expect);
+}
+
+#[tokio::test]
+async fn test_two_row_groups_with_all_nulls_in_one() {
+ let row_per_group = 5;
+ let reader = parquet_file(4, -2, 2, row_per_group);
+
+ let statistics =
+ arrow_statistics::ParquetColumnStatistics::from_parquet_statistics(&reader)
+ .expect("getting statistics");
+
+ // only 1 column
+ assert_eq!(statistics.len(), 1, "expected 1 column");
+
+ let arrow_stats = parquet_stats_to_arrow(&DataType::Int64, &statistics[0]).unwrap();
+ println!("Arrow statistics:\n {arrow_stats:#?}");
+
+ // 2 row groups
+ let mins = arrow_stats.min();
+ assert_eq!(mins.len(), 2, "expected 2 row groups");
+
+ // mins are [-2, null]
+ assert!(mins.is_null(1));
+ // check the first value -2
+ // let expect = Int64Array::from(vec![-2]);
+ // let expect = Arc::new(expect) as ArrayRef;
+ // assert_eq!(mins, &expect);
Review Comment:
It is getting late and I am tired. I will need to figure out how to get the first value to verify it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: API for collecting statistics/index for metadata of a parquet file + tests [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #10537:
URL: https://github.com/apache/datafusion/pull/10537
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: some tests to write data to a parquet file and read its metadata [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1602292079
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,528 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::sync::Arc;
+
+// use arrow::json::reader;
+use arrow_array::{make_array, Array, Int64Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+use parquet::file::statistics::Statistics as ParquetFileStatistics;
+use parquet::format::Statistics as ParquetFormatStatistics;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// TODO: Eventually, we will need only read_statistics and read_row_count,
+// but for my understading, we will keep both write and read statistics.
+// Their values should be the same
+pub struct RowGroupsStatistics {
+ // Statistics from the file writer
+ pub write_statistics: Vec<WriteRowGroupStatistics>,
+
+ // Statistics from the file reader
+ pub read_statistics: Vec<ReadRowGroupStatistics>,
+}
+
+pub struct WriteRowGroupStatistics {
+ pub statistics: ParquetFormatStatistics,
+ pub row_count: i64,
+}
+
+pub struct ReadRowGroupStatistics {
+ pub statistics: ParquetFileStatistics,
+ pub row_count: i64,
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_row_group_statistics(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> RowGroupsStatistics {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![
+ make_int64_batches_with_null(num_null, no_null_values_start, no_null_values_end), // TODO: likely make this more general using
+ // create_data_batch(scenario); where scenario is the respective enum data type
+ ];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ //////////////// WRITE STATISTICS ///////////////////////
+ let file_meta = writer.close().unwrap();
Review Comment:
With my surprise, the write and read statistics even though have the same content are stored in different structures. Write is `parquet::format::statistics` and read is `parquet::file::statistics::Statistics`. Why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: some tests to write data to a parquet file and read its metadata [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1603364922
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,528 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::sync::Arc;
+
+// use arrow::json::reader;
+use arrow_array::{make_array, Array, Int64Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+use parquet::file::statistics::Statistics as ParquetFileStatistics;
+use parquet::format::Statistics as ParquetFormatStatistics;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// TODO: Eventually, we will need only read_statistics and read_row_count,
+// but for my understading, we will keep both write and read statistics.
+// Their values should be the same
+pub struct RowGroupsStatistics {
+ // Statistics from the file writer
+ pub write_statistics: Vec<WriteRowGroupStatistics>,
+
+ // Statistics from the file reader
+ pub read_statistics: Vec<ReadRowGroupStatistics>,
+}
+
+pub struct WriteRowGroupStatistics {
+ pub statistics: ParquetFormatStatistics,
+ pub row_count: i64,
+}
+
+pub struct ReadRowGroupStatistics {
+ pub statistics: ParquetFileStatistics,
+ pub row_count: i64,
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_row_group_statistics(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> RowGroupsStatistics {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![
+ make_int64_batches_with_null(num_null, no_null_values_start, no_null_values_end), // TODO: likely make this more general using
+ // create_data_batch(scenario); where scenario is the respective enum data type
+ ];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ //////////////// WRITE STATISTICS ///////////////////////
+ let file_meta = writer.close().unwrap();
+
+ // meta data of the row groups
+ let rg_metas = file_meta.row_groups;
+
+ // Statstics of each grow group
+ let mut write_rg_statistics = Vec::new();
+ for rg_meta in rg_metas {
+ let column = &rg_meta.columns;
+ assert_eq!(column.len(), 1, "expected 1 column");
+ let col_metadata = &column[0].meta_data;
+ let col_metadata = col_metadata.as_ref().unwrap();
+ let stats = &col_metadata.statistics;
+ let stats = stats.as_ref().unwrap();
+
+ let rg_stats = WriteRowGroupStatistics {
+ row_count: rg_meta.num_rows,
+ statistics: stats.clone(),
+ };
+ write_rg_statistics.push(rg_stats);
+ }
+
+ //////////////// READ STATISTICS ///////////////////////
+ let file = output_file.reopen().unwrap();
+
+ let reader = ArrowReaderBuilder::try_new(file).unwrap();
+ let parquet_meta = reader.metadata().clone();
+
+ // meta data of the row groups
+ let rg_metas = parquet_meta.row_groups();
+
+ // Statstics of each grow group
+ let mut read_rg_statistics = Vec::new();
+ for rg_meta in rg_metas {
+ let column = &rg_meta.columns();
+ assert_eq!(column.len(), 1, "expected 1 column");
+ let stats = &column[0].statistics().unwrap();
+
+ let rg_stats = ReadRowGroupStatistics {
+ row_count: rg_meta.num_rows() as i64,
+ statistics: (*stats).clone(),
+ };
+ read_rg_statistics.push(rg_stats);
+ }
+
+ RowGroupsStatistics {
+ write_statistics: write_rg_statistics,
+ read_statistics: read_rg_statistics,
+ }
+}
+
+// TESTS
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let rg_statistics = parquet_row_group_statistics(0, 4, 7, row_per_group);
+
+ //////////// Verify write row group statistics ////////////
+ let write_rg_stats = &rg_statistics.write_statistics;
+
+ // only 1 row group
+ assert_eq!(write_rg_stats.len(), 1, "expected 1 row group");
+ let rg = &write_rg_stats[0];
+
+ // 3 rows
+ assert_eq!(rg.row_count, 3, "expected 3 rows");
+
+ // no nulls
+ assert!(rg.statistics.null_count.is_none(), "expected no nulls");
+
+ let statistics = &rg.statistics;
+
+ // min is [4, 0, 0, 0, 0, 0, 0, 0]
+ assert_eq!(
+ *statistics.min_value.as_ref().unwrap(),
+ [4, 0, 0, 0, 0, 0, 0, 0]
+ );
+ // max is [6, 0, 0, 0, 0, 0, 0, 0]
+ assert_eq!(
+ *statistics.max_value.as_ref().unwrap(),
+ [6, 0, 0, 0, 0, 0, 0, 0]
+ );
+
+ println!("RG statistics:\n {statistics:#?}");
+ // This will display:
+ // RG statistics:
+ // Statistics {
+ // max: Some(
+ // [
+ // 6,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // ],
+ // ),
+ // min: Some(
+ // [
+ // 4,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // ],
+ // ),
+ // null_count: None,
+ // distinct_count: None,
+ // max_value: Some(
+ // [
+ // 6,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // ],
+ // ),
+ // min_value: Some(
+ // [
+ // 4,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // ],
+ // ),
+ // is_max_value_exact: Some(
+ // true,
+ // ),
+ // is_min_value_exact: Some(
+ // true,
+ // ),
+ // }
+
+ //////////// Verify read row group statistics ////////////
+ let read_rg_stats = &rg_statistics.read_statistics;
+
+ // only 1 row group
+ assert_eq!(read_rg_stats.len(), 1, "expected 1 row group");
+ let rg = &read_rg_stats[0];
+
+ // 3 rows
+ assert_eq!(rg.row_count, 3, "expected 3 rows");
+
+ // rg.statistics is Int64
+ let stats = &rg.statistics;
+ match stats {
+ ParquetFileStatistics::Int64(int64_stats) => {
+ // no nulls
+ assert_eq!(int64_stats.null_count(), 0, "expected no nulls");
+ // min is 4
+ assert_eq!(*int64_stats.min(), 4);
+ // max is 6
+ assert_eq!(*int64_stats.max(), 6);
+ }
+ _ => panic!("expected Int64"),
+ }
Review Comment:
Rather than verifying the data that came out of the parquet reader, ideally here we would call `parquet_stats_to_arrow` and verify the output
For example, something like
```rust
// figure out the arrow datatype for "i64" column
let data_type = get_datatype_for_column("i64");
let stats = parquet_stats_to_arrow(
data_type,
rg.statistics.iter()
);
// verify that the min/max came out right
assert_eq!(stats.min, ArrayRef::new(Int32Array::from(vec![4, 0, 0]))));
// same for max
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: API for collecting statistics/index for metadata of a parquet file + tests [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1607104678
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,652 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::fs::File;
+use std::sync::Arc;
+
+use arrow_array::{
+ make_array, Array, ArrayRef, Int16Array, Int32Array, Int64Array, Int8Array,
+ RecordBatch, UInt64Array,
+};
+use arrow_schema::{DataType, Field, Schema};
+use datafusion::datasource::physical_plan::parquet::{
+ RequestedStatistics, StatisticsConverter,
+};
+use parquet::arrow::arrow_reader::{ArrowReaderBuilder, ParquetRecordBatchReaderBuilder};
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+
+use crate::parquet::Scenario;
+
+use super::make_test_file_rg;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_file_one_column(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![make_int64_batches_with_null(
+ num_null,
+ no_null_values_start,
+ no_null_values_end,
+ )];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ // close file
+ let _file_meta = writer.close().unwrap();
+
+ // open the file & get the reader
+ let file = output_file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+// Create a parquet file with many columns each has different data type
+// - Data types are specified by the given scenario
+// - Row group sizes are withe the same or different depending on the provided row_per_group & data created in the scenario
+pub async fn parquet_file_many_columns(
+ scenario: super::Scenario,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let file = make_test_file_rg(scenario, row_per_group).await;
+
+ // open the file & get the reader
+ let file = file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+struct Test {
+ reader: ParquetRecordBatchReaderBuilder<File>,
+ expected_min: ArrayRef,
+ expected_max: ArrayRef,
+ expected_null_counts: UInt64Array,
+ expected_row_counts: UInt64Array,
+}
+
+impl Test {
+ fn run(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min,
+ expected_max,
+ expected_null_counts,
+ expected_row_counts,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+
+ assert_eq!(&min, &expected_min, "Mismatch with expected minimums");
+
+ let max = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Max,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ assert_eq!(&max, &expected_max, "Mismatch with expected maximum");
+
+ let null_counts = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::NullCount,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ let expected_null_counts = Arc::new(expected_null_counts) as ArrayRef;
+ assert_eq!(
+ &null_counts, &expected_null_counts,
+ "Mismatch with expected null counts"
+ );
+
+ let row_counts = StatisticsConverter::row_counts(reader.metadata()).unwrap();
+ assert_eq!(
+ row_counts, expected_row_counts,
+ "Mismatch with expected row counts"
+ );
+ }
+
+ fn run_col_not_found(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min: _,
+ expected_max: _,
+ expected_null_counts: _,
+ expected_row_counts: _,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ );
+
+ assert!(min.is_err());
+ }
+}
+
+// TESTS
+//
+// Remaining cases
+// - Create parquet files / metadata with missing statistic values
+// - Create parquet files / metadata with different data types -- included but not all data types yet
+// - Create parquet files / metadata with different row group sizes -- done
+// - Using truncated statistics ("exact min value" and "exact max value" https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.max_is_exact)
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(0, 4, 7, row_per_group);
+ Test {
+ reader,
+ // min is 4
+ expected_min: Arc::new(Int64Array::from(vec![4])),
+ // max is 6
+ expected_max: Arc::new(Int64Array::from(vec![6])),
+ // no nulls
+ expected_null_counts: UInt64Array::from(vec![0]),
+ // 3 rows
+ expected_row_counts: UInt64Array::from(vec![3]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_one_row_group_with_null_and_negative() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(2, -1, 5, row_per_group);
+
+ Test {
+ reader,
+ // min is -1
+ expected_min: Arc::new(Int64Array::from(vec![-1])),
+ // max is 4
+ expected_max: Arc::new(Int64Array::from(vec![4])),
+ // 2 nulls
+ expected_null_counts: UInt64Array::from(vec![2]),
+ // 8 rows
+ expected_row_counts: UInt64Array::from(vec![8]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_group_with_null() {
+ let row_per_group = 10;
+ let reader = parquet_file_one_column(2, 4, 17, row_per_group);
+
+ Test {
+ reader,
+ // mins are [4, 14]
+ expected_min: Arc::new(Int64Array::from(vec![4, 14])),
+ // maxes are [13, 16]
+ expected_max: Arc::new(Int64Array::from(vec![13, 16])),
+ // nulls are [0, 2]
+ expected_null_counts: UInt64Array::from(vec![0, 2]),
+ // row counts are [10, 5]
+ expected_row_counts: UInt64Array::from(vec![10, 5]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_groups_with_all_nulls_in_one() {
+ let row_per_group = 5;
+ let reader = parquet_file_one_column(4, -2, 2, row_per_group);
+
+ Test {
+ reader,
+ // mins are [-2, null]
+ expected_min: Arc::new(Int64Array::from(vec![Some(-2), None])),
+ // maxes are [1, null]
+ expected_max: Arc::new(Int64Array::from(vec![Some(1), None])),
+ // nulls are [1, 3]
+ expected_null_counts: UInt64Array::from(vec![1, 3]),
+ // row counts are [5, 3]
+ expected_row_counts: UInt64Array::from(vec![5, 3]),
+ }
+ .run("i64")
+}
+
+/////////////// MORE GENERAL TESTS //////////////////////
+// . Many columns in a file
+// . Differnet data types
+// . Different row group sizes
+
+// Four different integer types
+#[tokio::test]
+async fn test_int_64() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i64");
+}
+
+#[tokio::test]
+async fn test_int_32() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int32Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int32Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i32");
+}
+
+// BUG: ignore this test for now
+// Note that the file has 4 columns named "i8", "i16", "i32", "i64".
+// - The tests on column i32 and i64 passed.
+// - The tests on column i8 and i16 failed.
+#[ignore]
+#[tokio::test]
+async fn test_int_16() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array type even though I debugged and the columns name is "i16" an its data is Int16
+ // My debugging tells me the bug is either at:
+ // 1. The new code to get "iter". See the code in this PR with
+ // // Get an iterator over the column statistics
+ // let iter = row_groups
+ // .iter()
+ // .map(|x| x.column(parquet_idx).statistics());
+ // OR
+ // 2. in the function (and/or its marco) `pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics>>>` here
+ // https://github.com/apache/datafusion/blob/ea023e2d4878240eece870cf4b346c7a0667aeed/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs#L179
+ expected_min: Arc::new(Int16Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int16Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i16");
+}
+
+// BUG (same as above): ignore this test for now
+#[ignore]
+#[tokio::test]
+async fn test_int_8() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array even though I debugged and the columns name is "i8" an its data is Int8
+ expected_min: Arc::new(Int8Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int8Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i8");
+}
+
+// timestamp
+#[tokio::test]
+async fn test_timestamp() {
+ let row_per_group = 5;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches, each has 5 rowws.
+ // Since the row group isze is set to 5, those 4 batches will go into 4 row groups
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577840471000000000, 1577841061000000000, 1578704461000000000,]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577840471000000000,
+ 1577841061000000000,
+ 1578704461000000000,
+ ])),
+ // maxes are [1577926861000000000, 1577926871000000000, 1577927461000000000, 1578790861000000000,]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1577926871000000000,
+ 1577927461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 1, 1, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577840471000000,
+ 1577841061000000,
+ 1578704461000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1577926871000000,
+ 1577927461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577840471000,
+ 1577841061000,
+ 1578704461000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1577926871000,
+ 1577927461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577840471, 1577841061, 1578704461,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1577926871, 1577927461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("seconds");
+}
+
+// timestamp with different row group sizes
+#[tokio::test]
+async fn test_timestamp_diff_rg_sizes() {
+ let row_per_group = 8;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 3 row groups with size 8, 8, 4
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577841061000000000, 1578704521000000000]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577841061000000000,
+ 1578704521000000000,
+ ])),
+ // maxes are [1577926861000000000, 1578704461000000000, 157879086100000000]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1578704461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 2, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ // row counts are [8, 8, 4]
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577841061000000,
+ 1578704521000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1578704461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577841061000,
+ 1578704521000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1578704461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577841061, 1578704521,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1578704461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("seconds");
+}
+
+// date with different row group sizes
+#[tokio::test]
+async fn test_dates_32_diff_rg_sizes() {
+ let row_per_group = 13;
+
+ // This creates a parquet files of 3 columns named "date32", "date64", "names"
+ // "date32" --> Date32Array
+ // "date64" --> Date64Array
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 2 row groups with size 13, 7
+ let reader = parquet_file_many_columns(Scenario::Dates, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [18262, 18565,]
+ expected_min: Arc::new(Int32Array::from(vec![18262, 18565])),
+ // maxes are [18564, 21865,]
+ expected_max: Arc::new(Int32Array::from(vec![18564, 21865])),
+ // nulls are [2, 2]
+ expected_null_counts: UInt64Array::from(vec![2, 2]),
+ // row counts are [13, 7]
+ expected_row_counts: UInt64Array::from(vec![13, 7]),
+ }
+ .run("date32");
+}
+
+// BUG: same as above. Expect to return Int64Array (for Date64Array) but returns Int32Array
+// test date with different row group sizes
+#[ignore]
+#[tokio::test]
+async fn test_dates_64_diff_rg_sizes() {
+ let row_per_group = 13;
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 2 row groups with size 13, 7
+ let reader = parquet_file_many_columns(Scenario::Dates, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![18262, 18565])), // panic here because the actual data is Int32Array
+ expected_max: Arc::new(Int64Array::from(vec![18564, 21865])),
+ expected_null_counts: UInt64Array::from(vec![2, 2]),
+ expected_row_counts: UInt64Array::from(vec![13, 7]),
+ }
+ .run("date64");
+}
+
+// TODO:
+// Other data types to tests
+// Int32Range,
Review Comment:
```suggestion
// `u8`, `u16`, `u32`, and `u64`,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: API for collecting statistics/index for metadata of a parquet file + tests [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1607118413
##########
datafusion/core/src/datasource/physical_plan/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,43 @@
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+use datafusion_common::Result;
+use parquet::file::statistics::Statistics as ParquetStatistics;
+
+/// statistics extracted from `Statistics` as Arrow `ArrayRef`s
+///
+/// # Note:
+/// If the corresponding `Statistics` is not present, or has no information for
+/// a column, a NULL is present in the corresponding array entry
+pub struct ArrowStatistics {
+ /// min values
+ min: ArrayRef,
+ /// max values
+ max: ArrayRef,
+ /// Row counts (UInt64Array)
+ row_count: ArrayRef,
+ /// Null Counts (UInt64Array)
+ null_count: ArrayRef,
+}
+
+/// Extract `ArrowStatistics` from the parquet [`Statistics`]
+pub fn parquet_stats_to_arrow<'a>(
+ arrow_datatype: &DataType,
+ statistics: impl IntoIterator<Item = Option<&'a ParquetStatistics>>,
+) -> Result<ArrowStatistics> {
+ todo!() // MY TODO next
+}
Review Comment:
Filed https://github.com/apache/datafusion/issues/10586
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: some tests to write data to a parquet file and read its metadata [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1604219819
##########
datafusion/core/src/datasource/physical_plan/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Extract parquet statistics and convert it to Arrow statistics
+
+use std::{fs::File, sync::Arc};
+
+use arrow_array::{ArrayRef, Int64Array, UInt64Array};
+use arrow_schema::DataType;
+use datafusion_common::{DataFusionError, Result};
+use parquet::{
+ arrow::arrow_reader::ParquetRecordBatchReaderBuilder,
+ file::statistics::{Statistics as ParquetStatistics, ValueStatistics},
+};
+
+use super::statistics::parquet_column;
+
+/// statistics extracted from `Statistics` as Arrow `ArrayRef`s
+///
+/// # Note:
+/// If the corresponding `Statistics` is not present, or has no information for
+/// a column, a NULL is present in the corresponding array entry
+#[derive(Debug)]
+pub struct ArrowStatistics {
+ /// min values
+ min: ArrayRef,
+ /// max values
+ max: ArrayRef,
+ /// Row counts (UInt64Array)
+ row_count: ArrayRef,
+ /// Null Counts (UInt64Array)
+ null_count: ArrayRef,
+}
+
+impl ArrowStatistics {
+ /// Create a new instance of `ArrowStatistics`
+ pub fn new(
+ min: ArrayRef,
+ max: ArrayRef,
+ row_count: ArrayRef,
+ null_count: ArrayRef,
+ ) -> Self {
+ Self {
+ min,
+ max,
+ row_count,
+ null_count,
+ }
+ }
+
+ /// Get the min values
+ pub fn min(&self) -> &ArrayRef {
+ &self.min
+ }
+
+ /// Get the max values
+ pub fn max(&self) -> &ArrayRef {
+ &self.max
+ }
+
+ /// Get the row counts
+ pub fn row_count(&self) -> &ArrayRef {
+ &self.row_count
+ }
+
+ /// Get the null counts
+ pub fn null_count(&self) -> &ArrayRef {
+ &self.null_count
+ }
+}
+
+/// Extract `ArrowStatistics` from the parquet [`Statistics`]
+pub fn parquet_stats_to_arrow(
+ arrow_datatype: &DataType,
+ statistics: &ParquetColumnStatistics,
+) -> Result<ArrowStatistics> {
+ // check of the data type is Int64
+ if !matches!(arrow_datatype, DataType::Int64) {
+ return Err(DataFusionError::Internal(format!(
+ "Unsupported data type {:?} for statistics",
+ arrow_datatype
+ )));
+ }
+
+ // row counts
+ let row_count = statistics
+ .rg_statistics
+ .iter()
+ .map(|rg| rg.row_count)
+ .collect::<Vec<_>>();
+
+ // get null counts
+ let parquet_stats = statistics.rg_statistics.iter().map(|rg| rg.statistics);
+ let null_counts = parquet_stats
+ // .map(|stats| stats.and_then(|s| Some(s.null_count())))
+ .map(|stats| stats.map(|s| s.null_count()))
+ .collect::<Vec<_>>();
+
+ // get min and max values
+ let parquet_stats_with_min_max = statistics
+ .rg_statistics
+ .iter()
+ .map(|rg| rg.get_statistics())
+ .collect::<Vec<_>>();
+
+ let mins = parquet_stats_with_min_max
+ .iter()
+ .map(|stats| {
+ stats.and_then(|s| {
+ let stats = ParquetColumnRowGroupStatistics::try_as_i64(s)?;
+ Some(*stats.min())
+ })
+ })
+ .collect::<Vec<_>>();
+
+ let maxs = parquet_stats_with_min_max
+ .iter()
+ .map(|stats| {
+ stats.and_then(|s| {
+ let stats = ParquetColumnRowGroupStatistics::try_as_i64(s)?;
+ Some(*stats.max())
+ })
+ })
+ .collect::<Vec<_>>();
+
+ Ok(ArrowStatistics {
+ min: Arc::new(Int64Array::from(mins)),
+ max: Arc::new(Int64Array::from(maxs)),
+ row_count: Arc::new(UInt64Array::from(row_count)),
+ null_count: Arc::new(UInt64Array::from(null_counts)),
+ })
+}
+
+/// All row group statistics of a file for a column
+pub struct ParquetColumnStatistics<'a> {
Review Comment:
I think this struct will be useful as intermediate steps to get only needed stats from parquet file
##########
datafusion/core/src/datasource/physical_plan/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Extract parquet statistics and convert it to Arrow statistics
+
+use std::{fs::File, sync::Arc};
+
+use arrow_array::{ArrayRef, Int64Array, UInt64Array};
+use arrow_schema::DataType;
+use datafusion_common::{DataFusionError, Result};
+use parquet::{
+ arrow::arrow_reader::ParquetRecordBatchReaderBuilder,
+ file::statistics::{Statistics as ParquetStatistics, ValueStatistics},
+};
+
+use super::statistics::parquet_column;
+
+/// statistics extracted from `Statistics` as Arrow `ArrayRef`s
+///
+/// # Note:
+/// If the corresponding `Statistics` is not present, or has no information for
+/// a column, a NULL is present in the corresponding array entry
+#[derive(Debug)]
+pub struct ArrowStatistics {
+ /// min values
+ min: ArrayRef,
+ /// max values
+ max: ArrayRef,
+ /// Row counts (UInt64Array)
+ row_count: ArrayRef,
+ /// Null Counts (UInt64Array)
+ null_count: ArrayRef,
+}
+
+impl ArrowStatistics {
+ /// Create a new instance of `ArrowStatistics`
+ pub fn new(
+ min: ArrayRef,
+ max: ArrayRef,
+ row_count: ArrayRef,
+ null_count: ArrayRef,
+ ) -> Self {
+ Self {
+ min,
+ max,
+ row_count,
+ null_count,
+ }
+ }
+
+ /// Get the min values
+ pub fn min(&self) -> &ArrayRef {
+ &self.min
+ }
+
+ /// Get the max values
+ pub fn max(&self) -> &ArrayRef {
+ &self.max
+ }
+
+ /// Get the row counts
+ pub fn row_count(&self) -> &ArrayRef {
+ &self.row_count
+ }
+
+ /// Get the null counts
+ pub fn null_count(&self) -> &ArrayRef {
+ &self.null_count
+ }
+}
+
+/// Extract `ArrowStatistics` from the parquet [`Statistics`]
+pub fn parquet_stats_to_arrow(
+ arrow_datatype: &DataType,
+ statistics: &ParquetColumnStatistics,
+) -> Result<ArrowStatistics> {
+ // check of the data type is Int64
+ if !matches!(arrow_datatype, DataType::Int64) {
+ return Err(DataFusionError::Internal(format!(
+ "Unsupported data type {:?} for statistics",
+ arrow_datatype
+ )));
+ }
+
+ // row counts
+ let row_count = statistics
+ .rg_statistics
+ .iter()
+ .map(|rg| rg.row_count)
+ .collect::<Vec<_>>();
+
+ // get null counts
+ let parquet_stats = statistics.rg_statistics.iter().map(|rg| rg.statistics);
+ let null_counts = parquet_stats
+ // .map(|stats| stats.and_then(|s| Some(s.null_count())))
+ .map(|stats| stats.map(|s| s.null_count()))
+ .collect::<Vec<_>>();
+
+ // get min and max values
+ let parquet_stats_with_min_max = statistics
+ .rg_statistics
+ .iter()
+ .map(|rg| rg.get_statistics())
+ .collect::<Vec<_>>();
+
+ let mins = parquet_stats_with_min_max
+ .iter()
+ .map(|stats| {
+ stats.and_then(|s| {
+ let stats = ParquetColumnRowGroupStatistics::try_as_i64(s)?;
+ Some(*stats.min())
+ })
+ })
+ .collect::<Vec<_>>();
+
+ let maxs = parquet_stats_with_min_max
+ .iter()
+ .map(|stats| {
+ stats.and_then(|s| {
+ let stats = ParquetColumnRowGroupStatistics::try_as_i64(s)?;
+ Some(*stats.max())
+ })
+ })
+ .collect::<Vec<_>>();
+
+ Ok(ArrowStatistics {
+ min: Arc::new(Int64Array::from(mins)),
+ max: Arc::new(Int64Array::from(maxs)),
+ row_count: Arc::new(UInt64Array::from(row_count)),
+ null_count: Arc::new(UInt64Array::from(null_counts)),
+ })
+}
+
+/// All row group statistics of a file for a column
+pub struct ParquetColumnStatistics<'a> {
+ // todo: do we need this?
+ // arrow column schema
+ // column_schema: &'a FieldRef,
+ _column_name: &'a str, // todo: do we need this?
+ rg_statistics: Vec<ParquetColumnRowGroupStatistics<'a>>,
+}
+
+/// Row group statistics of a column
+pub struct ParquetColumnRowGroupStatistics<'a> {
+ row_count: u64,
+ statistics: Option<&'a ParquetStatistics>,
+}
+
+impl<'a> ParquetColumnRowGroupStatistics<'a> {
+ /// Create a new instance of `ParquetColumnRowGroupStatistics`
+ pub fn new(row_count: u64, statistics: Option<&'a ParquetStatistics>) -> Self {
+ Self {
+ row_count,
+ statistics,
+ }
+ }
+
+ /// Return statistics if it exists and has min max
+ /// Otherwise return None
+ pub fn get_statistics(&self) -> Option<&'a ParquetStatistics> {
+ let stats = self.statistics?;
+ if stats.has_min_max_set() {
+ Some(stats)
+ } else {
+ None
+ }
+ }
+
+ /// Return the statistics as ValuesStatistcs<i64> if the column is i64
+ /// Otherwise return None
+ fn try_as_i64(stats: &'a ParquetStatistics) -> Option<&'a ValueStatistics<i64>> {
+ if let parquet::file::statistics::Statistics::Int64(statistics) = stats {
+ Some(statistics)
+ } else {
+ None
+ }
+ }
+}
+
+impl<'a> ParquetColumnStatistics<'a> {
+ /// Create a new instance of `ParquetColumnStatistics`
+ pub fn new(
+ _column_name: &'a str,
+ rg_statistics: Vec<ParquetColumnRowGroupStatistics<'a>>,
+ ) -> Self {
+ Self {
+ _column_name,
+ rg_statistics,
+ }
+ }
+
+ /// Extract statistics of all columns from a parquet file metadata
+ pub fn from_parquet_statistics(
Review Comment:
I actually want to add unit tests for this function. Will do it tomorrow
##########
datafusion/core/src/datasource/physical_plan/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Extract parquet statistics and convert it to Arrow statistics
+
+use std::{fs::File, sync::Arc};
+
+use arrow_array::{ArrayRef, Int64Array, UInt64Array};
+use arrow_schema::DataType;
+use datafusion_common::{DataFusionError, Result};
+use parquet::{
+ arrow::arrow_reader::ParquetRecordBatchReaderBuilder,
+ file::statistics::{Statistics as ParquetStatistics, ValueStatistics},
+};
+
+use super::statistics::parquet_column;
+
+/// statistics extracted from `Statistics` as Arrow `ArrayRef`s
+///
+/// # Note:
+/// If the corresponding `Statistics` is not present, or has no information for
+/// a column, a NULL is present in the corresponding array entry
+#[derive(Debug)]
+pub struct ArrowStatistics {
+ /// min values
+ min: ArrayRef,
+ /// max values
+ max: ArrayRef,
+ /// Row counts (UInt64Array)
+ row_count: ArrayRef,
+ /// Null Counts (UInt64Array)
+ null_count: ArrayRef,
+}
+
+impl ArrowStatistics {
+ /// Create a new instance of `ArrowStatistics`
+ pub fn new(
+ min: ArrayRef,
+ max: ArrayRef,
+ row_count: ArrayRef,
+ null_count: ArrayRef,
+ ) -> Self {
+ Self {
+ min,
+ max,
+ row_count,
+ null_count,
+ }
+ }
+
+ /// Get the min values
+ pub fn min(&self) -> &ArrayRef {
+ &self.min
+ }
+
+ /// Get the max values
+ pub fn max(&self) -> &ArrayRef {
+ &self.max
+ }
+
+ /// Get the row counts
+ pub fn row_count(&self) -> &ArrayRef {
+ &self.row_count
+ }
+
+ /// Get the null counts
+ pub fn null_count(&self) -> &ArrayRef {
+ &self.null_count
+ }
+}
+
+/// Extract `ArrowStatistics` from the parquet [`Statistics`]
+pub fn parquet_stats_to_arrow(
+ arrow_datatype: &DataType,
+ statistics: &ParquetColumnStatistics,
+) -> Result<ArrowStatistics> {
+ // check of the data type is Int64
+ if !matches!(arrow_datatype, DataType::Int64) {
+ return Err(DataFusionError::Internal(format!(
+ "Unsupported data type {:?} for statistics",
+ arrow_datatype
+ )));
+ }
+
+ // row counts
+ let row_count = statistics
+ .rg_statistics
+ .iter()
+ .map(|rg| rg.row_count)
+ .collect::<Vec<_>>();
+
+ // get null counts
+ let parquet_stats = statistics.rg_statistics.iter().map(|rg| rg.statistics);
+ let null_counts = parquet_stats
+ // .map(|stats| stats.and_then(|s| Some(s.null_count())))
Review Comment:
```suggestion
```
##########
datafusion/core/src/datasource/physical_plan/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,252 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Extract parquet statistics and convert it to Arrow statistics
+
+use std::{fs::File, sync::Arc};
+
+use arrow_array::{ArrayRef, Int64Array, UInt64Array};
+use arrow_schema::DataType;
+use datafusion_common::{DataFusionError, Result};
+use parquet::{
+ arrow::arrow_reader::ParquetRecordBatchReaderBuilder,
+ file::statistics::{Statistics as ParquetStatistics, ValueStatistics},
+};
+
+use super::statistics::parquet_column;
+
+/// statistics extracted from `Statistics` as Arrow `ArrayRef`s
+///
+/// # Note:
+/// If the corresponding `Statistics` is not present, or has no information for
+/// a column, a NULL is present in the corresponding array entry
+#[derive(Debug)]
+pub struct ArrowStatistics {
+ /// min values
+ min: ArrayRef,
+ /// max values
+ max: ArrayRef,
+ /// Row counts (UInt64Array)
+ row_count: ArrayRef,
+ /// Null Counts (UInt64Array)
+ null_count: ArrayRef,
+}
+
+impl ArrowStatistics {
+ /// Create a new instance of `ArrowStatistics`
+ pub fn new(
+ min: ArrayRef,
+ max: ArrayRef,
+ row_count: ArrayRef,
+ null_count: ArrayRef,
+ ) -> Self {
+ Self {
+ min,
+ max,
+ row_count,
+ null_count,
+ }
+ }
+
+ /// Get the min values
+ pub fn min(&self) -> &ArrayRef {
+ &self.min
+ }
+
+ /// Get the max values
+ pub fn max(&self) -> &ArrayRef {
+ &self.max
+ }
+
+ /// Get the row counts
+ pub fn row_count(&self) -> &ArrayRef {
+ &self.row_count
+ }
+
+ /// Get the null counts
+ pub fn null_count(&self) -> &ArrayRef {
+ &self.null_count
+ }
+}
+
+/// Extract `ArrowStatistics` from the parquet [`Statistics`]
+pub fn parquet_stats_to_arrow(
+ arrow_datatype: &DataType,
+ statistics: &ParquetColumnStatistics,
+) -> Result<ArrowStatistics> {
+ // check of the data type is Int64
+ if !matches!(arrow_datatype, DataType::Int64) {
+ return Err(DataFusionError::Internal(format!(
+ "Unsupported data type {:?} for statistics",
+ arrow_datatype
+ )));
+ }
+
+ // row counts
+ let row_count = statistics
+ .rg_statistics
+ .iter()
+ .map(|rg| rg.row_count)
+ .collect::<Vec<_>>();
+
+ // get null counts
+ let parquet_stats = statistics.rg_statistics.iter().map(|rg| rg.statistics);
+ let null_counts = parquet_stats
+ // .map(|stats| stats.and_then(|s| Some(s.null_count())))
+ .map(|stats| stats.map(|s| s.null_count()))
+ .collect::<Vec<_>>();
+
+ // get min and max values
+ let parquet_stats_with_min_max = statistics
+ .rg_statistics
+ .iter()
+ .map(|rg| rg.get_statistics())
+ .collect::<Vec<_>>();
+
+ let mins = parquet_stats_with_min_max
+ .iter()
+ .map(|stats| {
+ stats.and_then(|s| {
+ let stats = ParquetColumnRowGroupStatistics::try_as_i64(s)?;
+ Some(*stats.min())
+ })
+ })
+ .collect::<Vec<_>>();
+
+ let maxs = parquet_stats_with_min_max
+ .iter()
+ .map(|stats| {
+ stats.and_then(|s| {
+ let stats = ParquetColumnRowGroupStatistics::try_as_i64(s)?;
+ Some(*stats.max())
+ })
+ })
+ .collect::<Vec<_>>();
+
+ Ok(ArrowStatistics {
+ min: Arc::new(Int64Array::from(mins)),
+ max: Arc::new(Int64Array::from(maxs)),
+ row_count: Arc::new(UInt64Array::from(row_count)),
+ null_count: Arc::new(UInt64Array::from(null_counts)),
+ })
+}
+
+/// All row group statistics of a file for a column
+pub struct ParquetColumnStatistics<'a> {
+ // todo: do we need this?
+ // arrow column schema
+ // column_schema: &'a FieldRef,
+ _column_name: &'a str, // todo: do we need this?
+ rg_statistics: Vec<ParquetColumnRowGroupStatistics<'a>>,
+}
+
+/// Row group statistics of a column
+pub struct ParquetColumnRowGroupStatistics<'a> {
+ row_count: u64,
+ statistics: Option<&'a ParquetStatistics>,
+}
+
+impl<'a> ParquetColumnRowGroupStatistics<'a> {
+ /// Create a new instance of `ParquetColumnRowGroupStatistics`
+ pub fn new(row_count: u64, statistics: Option<&'a ParquetStatistics>) -> Self {
+ Self {
+ row_count,
+ statistics,
+ }
+ }
+
+ /// Return statistics if it exists and has min max
+ /// Otherwise return None
+ pub fn get_statistics(&self) -> Option<&'a ParquetStatistics> {
+ let stats = self.statistics?;
+ if stats.has_min_max_set() {
+ Some(stats)
+ } else {
+ None
+ }
+ }
+
+ /// Return the statistics as ValuesStatistcs<i64> if the column is i64
+ /// Otherwise return None
+ fn try_as_i64(stats: &'a ParquetStatistics) -> Option<&'a ValueStatistics<i64>> {
Review Comment:
For this PR, we focus on Int64 only. This function will be refactored in following PR(s)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: some tests to write data to a parquet file and read its metadata [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1602290548
##########
datafusion/core/src/datasource/physical_plan/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,43 @@
+use arrow_array::ArrayRef;
Review Comment:
And will add paragraph for the licensing here too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: API for collecting statistics/index for metadata of a parquet file [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#issuecomment-2118417052
@alamb : The PR is ready for review. Some notes:
1. The tests in `datafusion/core/src/datasource/physical_plan/parquet/statistics.rs` are tests for different dat types. Instead of using exact test inputs that do not cover everything, I am reusing a general available test inputs. Since I have not added all data types (which we agreed to do in a follow-on PR to avoid too much code), I decided not to remove any tests from that file yet. Will remove them when I am happy with the test coverage.
2. I have a feeling there will be a lot of tests of we want to cover all data types. I am looking froward to your feedback of tests in this PR for what we should not add.
3. I found a bug (the same for 3 tests in 3 different data types). I did a quick investigation but not very clear the root cause yet. I will do it next week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: API for collecting statistics/index for metadata of a parquet file [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1607091830
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,652 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::fs::File;
+use std::sync::Arc;
+
+use arrow_array::{
+ make_array, Array, ArrayRef, Int16Array, Int32Array, Int64Array, Int8Array,
+ RecordBatch, UInt64Array,
+};
+use arrow_schema::{DataType, Field, Schema};
+use datafusion::datasource::physical_plan::parquet::{
+ RequestedStatistics, StatisticsConverter,
+};
+use parquet::arrow::arrow_reader::{ArrowReaderBuilder, ParquetRecordBatchReaderBuilder};
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+
+use crate::parquet::Scenario;
+
+use super::make_test_file_rg;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_file_one_column(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![make_int64_batches_with_null(
+ num_null,
+ no_null_values_start,
+ no_null_values_end,
+ )];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ // close file
+ let _file_meta = writer.close().unwrap();
+
+ // open the file & get the reader
+ let file = output_file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+// Create a parquet file with many columns each has different data type
+// - Data types are specified by the given scenario
+// - Row group sizes are withe the same or different depending on the provided row_per_group & data created in the scenario
+pub async fn parquet_file_many_columns(
+ scenario: super::Scenario,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let file = make_test_file_rg(scenario, row_per_group).await;
+
+ // open the file & get the reader
+ let file = file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+struct Test {
+ reader: ParquetRecordBatchReaderBuilder<File>,
+ expected_min: ArrayRef,
+ expected_max: ArrayRef,
+ expected_null_counts: UInt64Array,
+ expected_row_counts: UInt64Array,
+}
+
+impl Test {
+ fn run(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min,
+ expected_max,
+ expected_null_counts,
+ expected_row_counts,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+
+ assert_eq!(&min, &expected_min, "Mismatch with expected minimums");
+
+ let max = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Max,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ assert_eq!(&max, &expected_max, "Mismatch with expected maximum");
+
+ let null_counts = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::NullCount,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ let expected_null_counts = Arc::new(expected_null_counts) as ArrayRef;
+ assert_eq!(
+ &null_counts, &expected_null_counts,
+ "Mismatch with expected null counts"
+ );
+
+ let row_counts = StatisticsConverter::row_counts(reader.metadata()).unwrap();
+ assert_eq!(
+ row_counts, expected_row_counts,
+ "Mismatch with expected row counts"
+ );
+ }
+
+ fn run_col_not_found(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min: _,
+ expected_max: _,
+ expected_null_counts: _,
+ expected_row_counts: _,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ );
+
+ assert!(min.is_err());
+ }
+}
+
+// TESTS
+//
+// Remaining cases
+// - Create parquet files / metadata with missing statistic values
+// - Create parquet files / metadata with different data types -- included but not all data types yet
+// - Create parquet files / metadata with different row group sizes -- done
+// - Using truncated statistics ("exact min value" and "exact max value" https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.max_is_exact)
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(0, 4, 7, row_per_group);
+ Test {
+ reader,
+ // min is 4
+ expected_min: Arc::new(Int64Array::from(vec![4])),
+ // max is 6
+ expected_max: Arc::new(Int64Array::from(vec![6])),
+ // no nulls
+ expected_null_counts: UInt64Array::from(vec![0]),
+ // 3 rows
+ expected_row_counts: UInt64Array::from(vec![3]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_one_row_group_with_null_and_negative() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(2, -1, 5, row_per_group);
+
+ Test {
+ reader,
+ // min is -1
+ expected_min: Arc::new(Int64Array::from(vec![-1])),
+ // max is 4
+ expected_max: Arc::new(Int64Array::from(vec![4])),
+ // 2 nulls
+ expected_null_counts: UInt64Array::from(vec![2]),
+ // 8 rows
+ expected_row_counts: UInt64Array::from(vec![8]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_group_with_null() {
+ let row_per_group = 10;
+ let reader = parquet_file_one_column(2, 4, 17, row_per_group);
+
+ Test {
+ reader,
+ // mins are [4, 14]
+ expected_min: Arc::new(Int64Array::from(vec![4, 14])),
+ // maxes are [13, 16]
+ expected_max: Arc::new(Int64Array::from(vec![13, 16])),
+ // nulls are [0, 2]
+ expected_null_counts: UInt64Array::from(vec![0, 2]),
+ // row counts are [10, 5]
+ expected_row_counts: UInt64Array::from(vec![10, 5]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_groups_with_all_nulls_in_one() {
+ let row_per_group = 5;
+ let reader = parquet_file_one_column(4, -2, 2, row_per_group);
+
+ Test {
+ reader,
+ // mins are [-2, null]
+ expected_min: Arc::new(Int64Array::from(vec![Some(-2), None])),
+ // maxes are [1, null]
+ expected_max: Arc::new(Int64Array::from(vec![Some(1), None])),
+ // nulls are [1, 3]
+ expected_null_counts: UInt64Array::from(vec![1, 3]),
+ // row counts are [5, 3]
+ expected_row_counts: UInt64Array::from(vec![5, 3]),
+ }
+ .run("i64")
+}
+
+/////////////// MORE GENERAL TESTS //////////////////////
+// . Many columns in a file
+// . Differnet data types
+// . Different row group sizes
+
+// Four different integer types
+#[tokio::test]
+async fn test_int_64() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i64");
+}
+
+#[tokio::test]
+async fn test_int_32() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int32Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int32Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i32");
+}
+
+// BUG: ignore this test for now
+// Note that the file has 4 columns named "i8", "i16", "i32", "i64".
+// - The tests on column i32 and i64 passed.
+// - The tests on column i8 and i16 failed.
+#[ignore]
+#[tokio::test]
+async fn test_int_16() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array type even though I debugged and the columns name is "i16" an its data is Int16
+ // My debugging tells me the bug is either at:
+ // 1. The new code to get "iter". See the code in this PR with
+ // // Get an iterator over the column statistics
+ // let iter = row_groups
+ // .iter()
+ // .map(|x| x.column(parquet_idx).statistics());
+ // OR
+ // 2. in the function (and/or its marco) `pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics>>>` here
+ // https://github.com/apache/datafusion/blob/ea023e2d4878240eece870cf4b346c7a0667aeed/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs#L179
+ expected_min: Arc::new(Int16Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int16Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i16");
+}
+
+// BUG (same as above): ignore this test for now
+#[ignore]
+#[tokio::test]
+async fn test_int_8() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array even though I debugged and the columns name is "i8" an its data is Int8
+ expected_min: Arc::new(Int8Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int8Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i8");
+}
+
+// timestamp
+#[tokio::test]
+async fn test_timestamp() {
+ let row_per_group = 5;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches, each has 5 rowws.
+ // Since the row group isze is set to 5, those 4 batches will go into 4 row groups
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577840471000000000, 1577841061000000000, 1578704461000000000,]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577840471000000000,
+ 1577841061000000000,
+ 1578704461000000000,
+ ])),
+ // maxes are [1577926861000000000, 1577926871000000000, 1577927461000000000, 1578790861000000000,]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1577926871000000000,
+ 1577927461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 1, 1, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577840471000000,
+ 1577841061000000,
+ 1578704461000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1577926871000000,
+ 1577927461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577840471000,
+ 1577841061000,
+ 1578704461000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1577926871000,
+ 1577927461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577840471, 1577841061, 1578704461,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1577926871, 1577927461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("seconds");
+}
+
+// timestamp with different row group sizes
+#[tokio::test]
+async fn test_timestamp_diff_rg_sizes() {
+ let row_per_group = 8;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 3 row groups with size 8, 8, 4
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577841061000000000, 1578704521000000000]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577841061000000000,
+ 1578704521000000000,
+ ])),
+ // maxes are [1577926861000000000, 1578704461000000000, 157879086100000000]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1578704461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 2, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ // row counts are [8, 8, 4]
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577841061000000,
+ 1578704521000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1578704461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577841061000,
+ 1578704521000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1578704461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577841061, 1578704521,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1578704461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("seconds");
+}
+
+// date with different row group sizes
+#[tokio::test]
+async fn test_dates_32_diff_rg_sizes() {
+ let row_per_group = 13;
+
+ // This creates a parquet files of 3 columns named "date32", "date64", "names"
+ // "date32" --> Date32Array
+ // "date64" --> Date64Array
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 2 row groups with size 13, 7
+ let reader = parquet_file_many_columns(Scenario::Dates, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [18262, 18565,]
+ expected_min: Arc::new(Int32Array::from(vec![18262, 18565])),
+ // maxes are [18564, 21865,]
+ expected_max: Arc::new(Int32Array::from(vec![18564, 21865])),
+ // nulls are [2, 2]
+ expected_null_counts: UInt64Array::from(vec![2, 2]),
+ // row counts are [13, 7]
+ expected_row_counts: UInt64Array::from(vec![13, 7]),
+ }
+ .run("date32");
+}
+
+// BUG: same as above. Expect to return Int64Array (for Date64Array) but returns Int32Array
+// test date with different row group sizes
+#[ignore]
+#[tokio::test]
+async fn test_dates_64_diff_rg_sizes() {
+ let row_per_group = 13;
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 2 row groups with size 13, 7
+ let reader = parquet_file_many_columns(Scenario::Dates, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![18262, 18565])), // panic here because the actual data is Int32Array
+ expected_max: Arc::new(Int64Array::from(vec![18564, 21865])),
+ expected_null_counts: UInt64Array::from(vec![2, 2]),
+ expected_row_counts: UInt64Array::from(vec![13, 7]),
+ }
+ .run("date64");
+}
+
+// TODO:
+// Other data types to tests
+// Int32Range,
+// UInt,
+// UInt32Range,
+// Float64,
+// Decimal,
+// DecimalBloomFilterInt32,
+// DecimalBloomFilterInt64,
+// DecimalLargePrecision,
+// DecimalLargePrecisionBloomFilter,
+// ByteArray,
+// PeriodsInColumnNames,
+// WithNullValuesPageLevel,
+// WITHOUT Stats
+
+/////// NEGATIVE TESTS ///////
+// column not found
+#[tokio::test]
+async fn test_column_not_found() {
+ let row_per_group = 5;
+ let reader = parquet_file_many_columns(Scenario::Dates, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![18262, 18565])),
+ expected_max: Arc::new(Int64Array::from(vec![18564, 21865])),
+ expected_null_counts: UInt64Array::from(vec![2, 2]),
+ expected_row_counts: UInt64Array::from(vec![13, 7]),
+ }
+ .run_col_not_found("not_a_column");
Review Comment:
👍
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,652 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::fs::File;
+use std::sync::Arc;
+
+use arrow_array::{
+ make_array, Array, ArrayRef, Int16Array, Int32Array, Int64Array, Int8Array,
+ RecordBatch, UInt64Array,
+};
+use arrow_schema::{DataType, Field, Schema};
+use datafusion::datasource::physical_plan::parquet::{
+ RequestedStatistics, StatisticsConverter,
+};
+use parquet::arrow::arrow_reader::{ArrowReaderBuilder, ParquetRecordBatchReaderBuilder};
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+
+use crate::parquet::Scenario;
+
+use super::make_test_file_rg;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_file_one_column(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![make_int64_batches_with_null(
+ num_null,
+ no_null_values_start,
+ no_null_values_end,
+ )];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ // close file
+ let _file_meta = writer.close().unwrap();
+
+ // open the file & get the reader
+ let file = output_file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+// Create a parquet file with many columns each has different data type
+// - Data types are specified by the given scenario
+// - Row group sizes are withe the same or different depending on the provided row_per_group & data created in the scenario
+pub async fn parquet_file_many_columns(
+ scenario: super::Scenario,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let file = make_test_file_rg(scenario, row_per_group).await;
+
+ // open the file & get the reader
+ let file = file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+struct Test {
+ reader: ParquetRecordBatchReaderBuilder<File>,
+ expected_min: ArrayRef,
+ expected_max: ArrayRef,
+ expected_null_counts: UInt64Array,
+ expected_row_counts: UInt64Array,
+}
+
+impl Test {
+ fn run(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min,
+ expected_max,
+ expected_null_counts,
+ expected_row_counts,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+
+ assert_eq!(&min, &expected_min, "Mismatch with expected minimums");
+
+ let max = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Max,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ assert_eq!(&max, &expected_max, "Mismatch with expected maximum");
+
+ let null_counts = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::NullCount,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ let expected_null_counts = Arc::new(expected_null_counts) as ArrayRef;
+ assert_eq!(
+ &null_counts, &expected_null_counts,
+ "Mismatch with expected null counts"
+ );
+
+ let row_counts = StatisticsConverter::row_counts(reader.metadata()).unwrap();
+ assert_eq!(
+ row_counts, expected_row_counts,
+ "Mismatch with expected row counts"
+ );
+ }
+
+ fn run_col_not_found(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min: _,
+ expected_max: _,
+ expected_null_counts: _,
+ expected_row_counts: _,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ );
+
+ assert!(min.is_err());
+ }
+}
+
+// TESTS
+//
+// Remaining cases
+// - Create parquet files / metadata with missing statistic values
+// - Create parquet files / metadata with different data types -- included but not all data types yet
+// - Create parquet files / metadata with different row group sizes -- done
+// - Using truncated statistics ("exact min value" and "exact max value" https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.max_is_exact)
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(0, 4, 7, row_per_group);
+ Test {
+ reader,
+ // min is 4
+ expected_min: Arc::new(Int64Array::from(vec![4])),
+ // max is 6
+ expected_max: Arc::new(Int64Array::from(vec![6])),
+ // no nulls
+ expected_null_counts: UInt64Array::from(vec![0]),
+ // 3 rows
+ expected_row_counts: UInt64Array::from(vec![3]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_one_row_group_with_null_and_negative() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(2, -1, 5, row_per_group);
+
+ Test {
+ reader,
+ // min is -1
+ expected_min: Arc::new(Int64Array::from(vec![-1])),
+ // max is 4
+ expected_max: Arc::new(Int64Array::from(vec![4])),
+ // 2 nulls
+ expected_null_counts: UInt64Array::from(vec![2]),
+ // 8 rows
+ expected_row_counts: UInt64Array::from(vec![8]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_group_with_null() {
+ let row_per_group = 10;
+ let reader = parquet_file_one_column(2, 4, 17, row_per_group);
+
+ Test {
+ reader,
+ // mins are [4, 14]
+ expected_min: Arc::new(Int64Array::from(vec![4, 14])),
+ // maxes are [13, 16]
+ expected_max: Arc::new(Int64Array::from(vec![13, 16])),
+ // nulls are [0, 2]
+ expected_null_counts: UInt64Array::from(vec![0, 2]),
+ // row counts are [10, 5]
+ expected_row_counts: UInt64Array::from(vec![10, 5]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_groups_with_all_nulls_in_one() {
+ let row_per_group = 5;
+ let reader = parquet_file_one_column(4, -2, 2, row_per_group);
+
+ Test {
+ reader,
+ // mins are [-2, null]
+ expected_min: Arc::new(Int64Array::from(vec![Some(-2), None])),
+ // maxes are [1, null]
+ expected_max: Arc::new(Int64Array::from(vec![Some(1), None])),
+ // nulls are [1, 3]
+ expected_null_counts: UInt64Array::from(vec![1, 3]),
+ // row counts are [5, 3]
+ expected_row_counts: UInt64Array::from(vec![5, 3]),
+ }
+ .run("i64")
+}
+
+/////////////// MORE GENERAL TESTS //////////////////////
+// . Many columns in a file
+// . Differnet data types
+// . Different row group sizes
+
+// Four different integer types
+#[tokio::test]
+async fn test_int_64() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i64");
+}
+
+#[tokio::test]
+async fn test_int_32() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int32Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int32Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i32");
+}
+
+// BUG: ignore this test for now
+// Note that the file has 4 columns named "i8", "i16", "i32", "i64".
+// - The tests on column i32 and i64 passed.
+// - The tests on column i8 and i16 failed.
+#[ignore]
+#[tokio::test]
+async fn test_int_16() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array type even though I debugged and the columns name is "i16" an its data is Int16
+ // My debugging tells me the bug is either at:
+ // 1. The new code to get "iter". See the code in this PR with
+ // // Get an iterator over the column statistics
+ // let iter = row_groups
+ // .iter()
+ // .map(|x| x.column(parquet_idx).statistics());
+ // OR
+ // 2. in the function (and/or its marco) `pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics>>>` here
+ // https://github.com/apache/datafusion/blob/ea023e2d4878240eece870cf4b346c7a0667aeed/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs#L179
+ expected_min: Arc::new(Int16Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int16Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i16");
+}
+
+// BUG (same as above): ignore this test for now
+#[ignore]
+#[tokio::test]
+async fn test_int_8() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array even though I debugged and the columns name is "i8" an its data is Int8
+ expected_min: Arc::new(Int8Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int8Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i8");
+}
+
+// timestamp
+#[tokio::test]
+async fn test_timestamp() {
+ let row_per_group = 5;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches, each has 5 rowws.
+ // Since the row group isze is set to 5, those 4 batches will go into 4 row groups
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577840471000000000, 1577841061000000000, 1578704461000000000,]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577840471000000000,
+ 1577841061000000000,
+ 1578704461000000000,
+ ])),
+ // maxes are [1577926861000000000, 1577926871000000000, 1577927461000000000, 1578790861000000000,]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1577926871000000000,
+ 1577927461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 1, 1, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577840471000000,
+ 1577841061000000,
+ 1578704461000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1577926871000000,
+ 1577927461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577840471000,
+ 1577841061000,
+ 1578704461000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1577926871000,
+ 1577927461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577840471, 1577841061, 1578704461,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1577926871, 1577927461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("seconds");
+}
+
+// timestamp with different row group sizes
+#[tokio::test]
+async fn test_timestamp_diff_rg_sizes() {
+ let row_per_group = 8;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 3 row groups with size 8, 8, 4
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577841061000000000, 1578704521000000000]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577841061000000000,
+ 1578704521000000000,
+ ])),
+ // maxes are [1577926861000000000, 1578704461000000000, 157879086100000000]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1578704461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 2, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ // row counts are [8, 8, 4]
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577841061000000,
+ 1578704521000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1578704461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577841061000,
+ 1578704521000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1578704461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577841061, 1578704521,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1578704461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("seconds");
+}
+
+// date with different row group sizes
+#[tokio::test]
+async fn test_dates_32_diff_rg_sizes() {
+ let row_per_group = 13;
+
+ // This creates a parquet files of 3 columns named "date32", "date64", "names"
+ // "date32" --> Date32Array
+ // "date64" --> Date64Array
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 2 row groups with size 13, 7
+ let reader = parquet_file_many_columns(Scenario::Dates, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [18262, 18565,]
+ expected_min: Arc::new(Int32Array::from(vec![18262, 18565])),
Review Comment:
I would actually expect the returned type to be `Date32Array` as the underlying arrow type is Date32 -- I don't think we need to make this change as part of this PR
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,652 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::fs::File;
+use std::sync::Arc;
+
+use arrow_array::{
+ make_array, Array, ArrayRef, Int16Array, Int32Array, Int64Array, Int8Array,
+ RecordBatch, UInt64Array,
+};
+use arrow_schema::{DataType, Field, Schema};
+use datafusion::datasource::physical_plan::parquet::{
+ RequestedStatistics, StatisticsConverter,
+};
+use parquet::arrow::arrow_reader::{ArrowReaderBuilder, ParquetRecordBatchReaderBuilder};
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+
+use crate::parquet::Scenario;
+
+use super::make_test_file_rg;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_file_one_column(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![make_int64_batches_with_null(
+ num_null,
+ no_null_values_start,
+ no_null_values_end,
+ )];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ // close file
+ let _file_meta = writer.close().unwrap();
+
+ // open the file & get the reader
+ let file = output_file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+// Create a parquet file with many columns each has different data type
+// - Data types are specified by the given scenario
+// - Row group sizes are withe the same or different depending on the provided row_per_group & data created in the scenario
+pub async fn parquet_file_many_columns(
+ scenario: super::Scenario,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let file = make_test_file_rg(scenario, row_per_group).await;
+
+ // open the file & get the reader
+ let file = file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+struct Test {
+ reader: ParquetRecordBatchReaderBuilder<File>,
+ expected_min: ArrayRef,
+ expected_max: ArrayRef,
+ expected_null_counts: UInt64Array,
+ expected_row_counts: UInt64Array,
+}
+
+impl Test {
+ fn run(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min,
+ expected_max,
+ expected_null_counts,
+ expected_row_counts,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+
+ assert_eq!(&min, &expected_min, "Mismatch with expected minimums");
+
+ let max = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Max,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ assert_eq!(&max, &expected_max, "Mismatch with expected maximum");
+
+ let null_counts = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::NullCount,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ let expected_null_counts = Arc::new(expected_null_counts) as ArrayRef;
+ assert_eq!(
+ &null_counts, &expected_null_counts,
+ "Mismatch with expected null counts"
+ );
+
+ let row_counts = StatisticsConverter::row_counts(reader.metadata()).unwrap();
+ assert_eq!(
+ row_counts, expected_row_counts,
+ "Mismatch with expected row counts"
+ );
+ }
+
+ fn run_col_not_found(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min: _,
+ expected_max: _,
+ expected_null_counts: _,
+ expected_row_counts: _,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ );
+
+ assert!(min.is_err());
+ }
+}
+
+// TESTS
+//
+// Remaining cases
+// - Create parquet files / metadata with missing statistic values
+// - Create parquet files / metadata with different data types -- included but not all data types yet
+// - Create parquet files / metadata with different row group sizes -- done
+// - Using truncated statistics ("exact min value" and "exact max value" https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.max_is_exact)
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(0, 4, 7, row_per_group);
+ Test {
+ reader,
+ // min is 4
+ expected_min: Arc::new(Int64Array::from(vec![4])),
+ // max is 6
+ expected_max: Arc::new(Int64Array::from(vec![6])),
+ // no nulls
+ expected_null_counts: UInt64Array::from(vec![0]),
+ // 3 rows
+ expected_row_counts: UInt64Array::from(vec![3]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_one_row_group_with_null_and_negative() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(2, -1, 5, row_per_group);
+
+ Test {
+ reader,
+ // min is -1
+ expected_min: Arc::new(Int64Array::from(vec![-1])),
+ // max is 4
+ expected_max: Arc::new(Int64Array::from(vec![4])),
+ // 2 nulls
+ expected_null_counts: UInt64Array::from(vec![2]),
+ // 8 rows
+ expected_row_counts: UInt64Array::from(vec![8]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_group_with_null() {
+ let row_per_group = 10;
+ let reader = parquet_file_one_column(2, 4, 17, row_per_group);
+
+ Test {
+ reader,
+ // mins are [4, 14]
+ expected_min: Arc::new(Int64Array::from(vec![4, 14])),
+ // maxes are [13, 16]
+ expected_max: Arc::new(Int64Array::from(vec![13, 16])),
+ // nulls are [0, 2]
+ expected_null_counts: UInt64Array::from(vec![0, 2]),
+ // row counts are [10, 5]
+ expected_row_counts: UInt64Array::from(vec![10, 5]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_groups_with_all_nulls_in_one() {
+ let row_per_group = 5;
+ let reader = parquet_file_one_column(4, -2, 2, row_per_group);
+
+ Test {
+ reader,
+ // mins are [-2, null]
+ expected_min: Arc::new(Int64Array::from(vec![Some(-2), None])),
+ // maxes are [1, null]
+ expected_max: Arc::new(Int64Array::from(vec![Some(1), None])),
+ // nulls are [1, 3]
+ expected_null_counts: UInt64Array::from(vec![1, 3]),
+ // row counts are [5, 3]
+ expected_row_counts: UInt64Array::from(vec![5, 3]),
+ }
+ .run("i64")
+}
+
+/////////////// MORE GENERAL TESTS //////////////////////
+// . Many columns in a file
+// . Differnet data types
+// . Different row group sizes
+
+// Four different integer types
+#[tokio::test]
+async fn test_int_64() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i64");
+}
+
+#[tokio::test]
+async fn test_int_32() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int32Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int32Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i32");
+}
+
+// BUG: ignore this test for now
+// Note that the file has 4 columns named "i8", "i16", "i32", "i64".
+// - The tests on column i32 and i64 passed.
+// - The tests on column i8 and i16 failed.
+#[ignore]
+#[tokio::test]
+async fn test_int_16() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array type even though I debugged and the columns name is "i16" an its data is Int16
+ // My debugging tells me the bug is either at:
+ // 1. The new code to get "iter". See the code in this PR with
+ // // Get an iterator over the column statistics
+ // let iter = row_groups
+ // .iter()
+ // .map(|x| x.column(parquet_idx).statistics());
+ // OR
+ // 2. in the function (and/or its marco) `pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics>>>` here
+ // https://github.com/apache/datafusion/blob/ea023e2d4878240eece870cf4b346c7a0667aeed/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs#L179
+ expected_min: Arc::new(Int16Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int16Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i16");
+}
+
+// BUG (same as above): ignore this test for now
+#[ignore]
+#[tokio::test]
+async fn test_int_8() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array even though I debugged and the columns name is "i8" an its data is Int8
+ expected_min: Arc::new(Int8Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int8Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i8");
+}
+
+// timestamp
+#[tokio::test]
+async fn test_timestamp() {
+ let row_per_group = 5;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches, each has 5 rowws.
+ // Since the row group isze is set to 5, those 4 batches will go into 4 row groups
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577840471000000000, 1577841061000000000, 1578704461000000000,]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577840471000000000,
+ 1577841061000000000,
+ 1578704461000000000,
+ ])),
+ // maxes are [1577926861000000000, 1577926871000000000, 1577927461000000000, 1578790861000000000,]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1577926871000000000,
+ 1577927461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 1, 1, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577840471000000,
+ 1577841061000000,
+ 1578704461000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1577926871000000,
+ 1577927461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577840471000,
+ 1577841061000,
+ 1578704461000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1577926871000,
+ 1577927461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577840471, 1577841061, 1578704461,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1577926871, 1577927461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("seconds");
+}
+
+// timestamp with different row group sizes
+#[tokio::test]
+async fn test_timestamp_diff_rg_sizes() {
+ let row_per_group = 8;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 3 row groups with size 8, 8, 4
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577841061000000000, 1578704521000000000]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577841061000000000,
+ 1578704521000000000,
+ ])),
+ // maxes are [1577926861000000000, 1578704461000000000, 157879086100000000]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1578704461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 2, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ // row counts are [8, 8, 4]
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577841061000000,
+ 1578704521000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1578704461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577841061000,
+ 1578704521000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1578704461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577841061, 1578704521,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1578704461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("seconds");
+}
+
+// date with different row group sizes
+#[tokio::test]
+async fn test_dates_32_diff_rg_sizes() {
+ let row_per_group = 13;
+
+ // This creates a parquet files of 3 columns named "date32", "date64", "names"
+ // "date32" --> Date32Array
+ // "date64" --> Date64Array
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 2 row groups with size 13, 7
+ let reader = parquet_file_many_columns(Scenario::Dates, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [18262, 18565,]
+ expected_min: Arc::new(Int32Array::from(vec![18262, 18565])),
+ // maxes are [18564, 21865,]
+ expected_max: Arc::new(Int32Array::from(vec![18564, 21865])),
+ // nulls are [2, 2]
+ expected_null_counts: UInt64Array::from(vec![2, 2]),
+ // row counts are [13, 7]
+ expected_row_counts: UInt64Array::from(vec![13, 7]),
+ }
+ .run("date32");
+}
+
+// BUG: same as above. Expect to return Int64Array (for Date64Array) but returns Int32Array
+// test date with different row group sizes
+#[ignore]
+#[tokio::test]
+async fn test_dates_64_diff_rg_sizes() {
+ let row_per_group = 13;
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 2 row groups with size 13, 7
+ let reader = parquet_file_many_columns(Scenario::Dates, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![18262, 18565])), // panic here because the actual data is Int32Array
Review Comment:
Likewise, I would expect this to be Date64Array (not Int64Array).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: API for collecting statistics/index for metadata of a parquet file + tests [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1607125587
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,654 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::fs::File;
+use std::sync::Arc;
+
+use arrow_array::{
+ make_array, Array, ArrayRef, Int16Array, Int32Array, Int64Array, Int8Array,
+ RecordBatch, UInt64Array,
+};
+use arrow_schema::{DataType, Field, Schema};
+use datafusion::datasource::physical_plan::parquet::{
+ RequestedStatistics, StatisticsConverter,
+};
+use parquet::arrow::arrow_reader::{ArrowReaderBuilder, ParquetRecordBatchReaderBuilder};
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+
+use crate::parquet::Scenario;
+
+use super::make_test_file_rg;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_file_one_column(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![make_int64_batches_with_null(
+ num_null,
+ no_null_values_start,
+ no_null_values_end,
+ )];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ // close file
+ let _file_meta = writer.close().unwrap();
+
+ // open the file & get the reader
+ let file = output_file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+// Create a parquet file with many columns each has different data type
+// - Data types are specified by the given scenario
+// - Row group sizes are withe the same or different depending on the provided row_per_group & data created in the scenario
+pub async fn parquet_file_many_columns(
+ scenario: super::Scenario,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let file = make_test_file_rg(scenario, row_per_group).await;
+
+ // open the file & get the reader
+ let file = file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+struct Test {
+ reader: ParquetRecordBatchReaderBuilder<File>,
+ expected_min: ArrayRef,
+ expected_max: ArrayRef,
+ expected_null_counts: UInt64Array,
+ expected_row_counts: UInt64Array,
+}
+
+impl Test {
+ fn run(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min,
+ expected_max,
+ expected_null_counts,
+ expected_row_counts,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+
+ assert_eq!(&min, &expected_min, "Mismatch with expected minimums");
+
+ let max = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Max,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ assert_eq!(&max, &expected_max, "Mismatch with expected maximum");
+
+ let null_counts = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::NullCount,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ let expected_null_counts = Arc::new(expected_null_counts) as ArrayRef;
+ assert_eq!(
+ &null_counts, &expected_null_counts,
+ "Mismatch with expected null counts"
+ );
+
+ let row_counts = StatisticsConverter::row_counts(reader.metadata()).unwrap();
+ assert_eq!(
+ row_counts, expected_row_counts,
+ "Mismatch with expected row counts"
+ );
+ }
+
+ fn run_col_not_found(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min: _,
+ expected_max: _,
+ expected_null_counts: _,
+ expected_row_counts: _,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ );
+
+ assert!(min.is_err());
+ }
+}
+
+// TESTS
+//
+// Remaining cases
+// - Create parquet files / metadata with missing statistic values
+// - Create parquet files / metadata with different data types -- included but not all data types yet
+// - Create parquet files / metadata with different row group sizes -- done
+// - Using truncated statistics ("exact min value" and "exact max value" https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.max_is_exact)
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(0, 4, 7, row_per_group);
+ Test {
+ reader,
+ // min is 4
+ expected_min: Arc::new(Int64Array::from(vec![4])),
+ // max is 6
+ expected_max: Arc::new(Int64Array::from(vec![6])),
+ // no nulls
+ expected_null_counts: UInt64Array::from(vec![0]),
+ // 3 rows
+ expected_row_counts: UInt64Array::from(vec![3]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_one_row_group_with_null_and_negative() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(2, -1, 5, row_per_group);
+
+ Test {
+ reader,
+ // min is -1
+ expected_min: Arc::new(Int64Array::from(vec![-1])),
+ // max is 4
+ expected_max: Arc::new(Int64Array::from(vec![4])),
+ // 2 nulls
+ expected_null_counts: UInt64Array::from(vec![2]),
+ // 8 rows
+ expected_row_counts: UInt64Array::from(vec![8]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_group_with_null() {
+ let row_per_group = 10;
+ let reader = parquet_file_one_column(2, 4, 17, row_per_group);
+
+ Test {
+ reader,
+ // mins are [4, 14]
+ expected_min: Arc::new(Int64Array::from(vec![4, 14])),
+ // maxes are [13, 16]
+ expected_max: Arc::new(Int64Array::from(vec![13, 16])),
+ // nulls are [0, 2]
+ expected_null_counts: UInt64Array::from(vec![0, 2]),
+ // row counts are [10, 5]
+ expected_row_counts: UInt64Array::from(vec![10, 5]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_groups_with_all_nulls_in_one() {
+ let row_per_group = 5;
+ let reader = parquet_file_one_column(4, -2, 2, row_per_group);
+
+ Test {
+ reader,
+ // mins are [-2, null]
+ expected_min: Arc::new(Int64Array::from(vec![Some(-2), None])),
+ // maxes are [1, null]
+ expected_max: Arc::new(Int64Array::from(vec![Some(1), None])),
+ // nulls are [1, 3]
+ expected_null_counts: UInt64Array::from(vec![1, 3]),
+ // row counts are [5, 3]
+ expected_row_counts: UInt64Array::from(vec![5, 3]),
+ }
+ .run("i64")
+}
+
+/////////////// MORE GENERAL TESTS //////////////////////
+// . Many columns in a file
+// . Differnet data types
+// . Different row group sizes
+
+// Four different integer types
+#[tokio::test]
+async fn test_int_64() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i64");
+}
+
+#[tokio::test]
+async fn test_int_32() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int32Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int32Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i32");
+}
+
+// BUG: ignore this test for now
+// https://github.com/apache/datafusion/issues/10585
+// Note that the file has 4 columns named "i8", "i16", "i32", "i64".
+// - The tests on column i32 and i64 passed.
+// - The tests on column i8 and i16 failed.
+#[ignore]
+#[tokio::test]
+async fn test_int_16() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array type even though I debugged and the columns name is "i16" an its data is Int16
+ // My debugging tells me the bug is either at:
+ // 1. The new code to get "iter". See the code in this PR with
+ // // Get an iterator over the column statistics
+ // let iter = row_groups
+ // .iter()
+ // .map(|x| x.column(parquet_idx).statistics());
+ // OR
+ // 2. in the function (and/or its marco) `pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics>>>` here
+ // https://github.com/apache/datafusion/blob/ea023e2d4878240eece870cf4b346c7a0667aeed/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs#L179
+ expected_min: Arc::new(Int16Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int16Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i16");
+}
+
+// BUG (same as above): ignore this test for now
+// https://github.com/apache/datafusion/issues/10585
+#[ignore]
+#[tokio::test]
+async fn test_int_8() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array even though I debugged and the columns name is "i8" an its data is Int8
+ expected_min: Arc::new(Int8Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int8Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i8");
+}
+
+// timestamp
+#[tokio::test]
+async fn test_timestamp() {
+ let row_per_group = 5;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches, each has 5 rowws.
+ // Since the row group isze is set to 5, those 4 batches will go into 4 row groups
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577840471000000000, 1577841061000000000, 1578704461000000000,]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577840471000000000,
+ 1577841061000000000,
+ 1578704461000000000,
+ ])),
+ // maxes are [1577926861000000000, 1577926871000000000, 1577927461000000000, 1578790861000000000,]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1577926871000000000,
+ 1577927461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 1, 1, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577840471000000,
+ 1577841061000000,
+ 1578704461000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1577926871000000,
+ 1577927461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577840471000,
+ 1577841061000,
+ 1578704461000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1577926871000,
+ 1577927461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577840471, 1577841061, 1578704461,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1577926871, 1577927461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("seconds");
+}
+
+// timestamp with different row group sizes
+#[tokio::test]
+async fn test_timestamp_diff_rg_sizes() {
+ let row_per_group = 8;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 3 row groups with size 8, 8, 4
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577841061000000000, 1578704521000000000]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577841061000000000,
+ 1578704521000000000,
+ ])),
+ // maxes are [1577926861000000000, 1578704461000000000, 157879086100000000]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1578704461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 2, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ // row counts are [8, 8, 4]
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577841061000000,
+ 1578704521000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1578704461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577841061000,
+ 1578704521000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1578704461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577841061, 1578704521,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1578704461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("seconds");
+}
+
+// date with different row group sizes
+#[tokio::test]
+async fn test_dates_32_diff_rg_sizes() {
+ let row_per_group = 13;
+
+ // This creates a parquet files of 3 columns named "date32", "date64", "names"
+ // "date32" --> Date32Array
+ // "date64" --> Date64Array
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 2 row groups with size 13, 7
+ let reader = parquet_file_many_columns(Scenario::Dates, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [18262, 18565,]
+ expected_min: Arc::new(Int32Array::from(vec![18262, 18565])),
+ // maxes are [18564, 21865,]
+ expected_max: Arc::new(Int32Array::from(vec![18564, 21865])),
+ // nulls are [2, 2]
+ expected_null_counts: UInt64Array::from(vec![2, 2]),
+ // row counts are [13, 7]
+ expected_row_counts: UInt64Array::from(vec![13, 7]),
+ }
+ .run("date32");
+}
+
+// BUG: same as above. Expect to return Int64Array (for Date64Array) but returns Int32Array
+// test date with different row group sizes
Review Comment:
```suggestion
// BUG: same as above. Expect to return Date64Array but returns Int32Array
// test date with different row group sizes
// https://github.com/apache/datafusion/issues/10587
```
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,654 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::fs::File;
+use std::sync::Arc;
+
+use arrow_array::{
+ make_array, Array, ArrayRef, Int16Array, Int32Array, Int64Array, Int8Array,
+ RecordBatch, UInt64Array,
+};
+use arrow_schema::{DataType, Field, Schema};
+use datafusion::datasource::physical_plan::parquet::{
+ RequestedStatistics, StatisticsConverter,
+};
+use parquet::arrow::arrow_reader::{ArrowReaderBuilder, ParquetRecordBatchReaderBuilder};
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+
+use crate::parquet::Scenario;
+
+use super::make_test_file_rg;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_file_one_column(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![make_int64_batches_with_null(
+ num_null,
+ no_null_values_start,
+ no_null_values_end,
+ )];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ // close file
+ let _file_meta = writer.close().unwrap();
+
+ // open the file & get the reader
+ let file = output_file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+// Create a parquet file with many columns each has different data type
+// - Data types are specified by the given scenario
+// - Row group sizes are withe the same or different depending on the provided row_per_group & data created in the scenario
+pub async fn parquet_file_many_columns(
+ scenario: super::Scenario,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let file = make_test_file_rg(scenario, row_per_group).await;
+
+ // open the file & get the reader
+ let file = file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+struct Test {
+ reader: ParquetRecordBatchReaderBuilder<File>,
+ expected_min: ArrayRef,
+ expected_max: ArrayRef,
+ expected_null_counts: UInt64Array,
+ expected_row_counts: UInt64Array,
+}
+
+impl Test {
+ fn run(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min,
+ expected_max,
+ expected_null_counts,
+ expected_row_counts,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+
+ assert_eq!(&min, &expected_min, "Mismatch with expected minimums");
+
+ let max = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Max,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ assert_eq!(&max, &expected_max, "Mismatch with expected maximum");
+
+ let null_counts = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::NullCount,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ let expected_null_counts = Arc::new(expected_null_counts) as ArrayRef;
+ assert_eq!(
+ &null_counts, &expected_null_counts,
+ "Mismatch with expected null counts"
+ );
+
+ let row_counts = StatisticsConverter::row_counts(reader.metadata()).unwrap();
+ assert_eq!(
+ row_counts, expected_row_counts,
+ "Mismatch with expected row counts"
+ );
+ }
+
+ fn run_col_not_found(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min: _,
+ expected_max: _,
+ expected_null_counts: _,
+ expected_row_counts: _,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ );
+
+ assert!(min.is_err());
+ }
+}
+
+// TESTS
+//
+// Remaining cases
+// - Create parquet files / metadata with missing statistic values
+// - Create parquet files / metadata with different data types -- included but not all data types yet
+// - Create parquet files / metadata with different row group sizes -- done
+// - Using truncated statistics ("exact min value" and "exact max value" https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.max_is_exact)
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(0, 4, 7, row_per_group);
+ Test {
+ reader,
+ // min is 4
+ expected_min: Arc::new(Int64Array::from(vec![4])),
+ // max is 6
+ expected_max: Arc::new(Int64Array::from(vec![6])),
+ // no nulls
+ expected_null_counts: UInt64Array::from(vec![0]),
+ // 3 rows
+ expected_row_counts: UInt64Array::from(vec![3]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_one_row_group_with_null_and_negative() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(2, -1, 5, row_per_group);
+
+ Test {
+ reader,
+ // min is -1
+ expected_min: Arc::new(Int64Array::from(vec![-1])),
+ // max is 4
+ expected_max: Arc::new(Int64Array::from(vec![4])),
+ // 2 nulls
+ expected_null_counts: UInt64Array::from(vec![2]),
+ // 8 rows
+ expected_row_counts: UInt64Array::from(vec![8]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_group_with_null() {
+ let row_per_group = 10;
+ let reader = parquet_file_one_column(2, 4, 17, row_per_group);
+
+ Test {
+ reader,
+ // mins are [4, 14]
+ expected_min: Arc::new(Int64Array::from(vec![4, 14])),
+ // maxes are [13, 16]
+ expected_max: Arc::new(Int64Array::from(vec![13, 16])),
+ // nulls are [0, 2]
+ expected_null_counts: UInt64Array::from(vec![0, 2]),
+ // row counts are [10, 5]
+ expected_row_counts: UInt64Array::from(vec![10, 5]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_groups_with_all_nulls_in_one() {
+ let row_per_group = 5;
+ let reader = parquet_file_one_column(4, -2, 2, row_per_group);
+
+ Test {
+ reader,
+ // mins are [-2, null]
+ expected_min: Arc::new(Int64Array::from(vec![Some(-2), None])),
+ // maxes are [1, null]
+ expected_max: Arc::new(Int64Array::from(vec![Some(1), None])),
+ // nulls are [1, 3]
+ expected_null_counts: UInt64Array::from(vec![1, 3]),
+ // row counts are [5, 3]
+ expected_row_counts: UInt64Array::from(vec![5, 3]),
+ }
+ .run("i64")
+}
+
+/////////////// MORE GENERAL TESTS //////////////////////
+// . Many columns in a file
+// . Differnet data types
+// . Different row group sizes
+
+// Four different integer types
+#[tokio::test]
+async fn test_int_64() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i64");
+}
+
+#[tokio::test]
+async fn test_int_32() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int32Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int32Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i32");
+}
+
+// BUG: ignore this test for now
+// https://github.com/apache/datafusion/issues/10585
+// Note that the file has 4 columns named "i8", "i16", "i32", "i64".
+// - The tests on column i32 and i64 passed.
+// - The tests on column i8 and i16 failed.
+#[ignore]
+#[tokio::test]
+async fn test_int_16() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array type even though I debugged and the columns name is "i16" an its data is Int16
+ // My debugging tells me the bug is either at:
+ // 1. The new code to get "iter". See the code in this PR with
+ // // Get an iterator over the column statistics
+ // let iter = row_groups
+ // .iter()
+ // .map(|x| x.column(parquet_idx).statistics());
+ // OR
+ // 2. in the function (and/or its marco) `pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics>>>` here
+ // https://github.com/apache/datafusion/blob/ea023e2d4878240eece870cf4b346c7a0667aeed/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs#L179
+ expected_min: Arc::new(Int16Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int16Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i16");
+}
+
+// BUG (same as above): ignore this test for now
+// https://github.com/apache/datafusion/issues/10585
+#[ignore]
+#[tokio::test]
+async fn test_int_8() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array even though I debugged and the columns name is "i8" an its data is Int8
+ expected_min: Arc::new(Int8Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int8Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i8");
+}
+
+// timestamp
+#[tokio::test]
+async fn test_timestamp() {
+ let row_per_group = 5;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches, each has 5 rowws.
+ // Since the row group isze is set to 5, those 4 batches will go into 4 row groups
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577840471000000000, 1577841061000000000, 1578704461000000000,]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577840471000000000,
+ 1577841061000000000,
+ 1578704461000000000,
+ ])),
+ // maxes are [1577926861000000000, 1577926871000000000, 1577927461000000000, 1578790861000000000,]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1577926871000000000,
+ 1577927461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 1, 1, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577840471000000,
+ 1577841061000000,
+ 1578704461000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1577926871000000,
+ 1577927461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577840471000,
+ 1577841061000,
+ 1578704461000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1577926871000,
+ 1577927461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577840471, 1577841061, 1578704461,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1577926871, 1577927461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("seconds");
+}
+
+// timestamp with different row group sizes
+#[tokio::test]
+async fn test_timestamp_diff_rg_sizes() {
+ let row_per_group = 8;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 3 row groups with size 8, 8, 4
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577841061000000000, 1578704521000000000]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577841061000000000,
+ 1578704521000000000,
+ ])),
+ // maxes are [1577926861000000000, 1578704461000000000, 157879086100000000]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1578704461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 2, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ // row counts are [8, 8, 4]
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577841061000000,
+ 1578704521000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1578704461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577841061000,
+ 1578704521000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1578704461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577841061, 1578704521,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1578704461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("seconds");
+}
+
+// date with different row group sizes
Review Comment:
```suggestion
// date with different row group sizes
// Bug expect `Date32Array` but returns Int32Array
// https://github.com/apache/datafusion/issues/10587
```
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,652 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::fs::File;
+use std::sync::Arc;
+
+use arrow_array::{
+ make_array, Array, ArrayRef, Int16Array, Int32Array, Int64Array, Int8Array,
+ RecordBatch, UInt64Array,
+};
+use arrow_schema::{DataType, Field, Schema};
+use datafusion::datasource::physical_plan::parquet::{
+ RequestedStatistics, StatisticsConverter,
+};
+use parquet::arrow::arrow_reader::{ArrowReaderBuilder, ParquetRecordBatchReaderBuilder};
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+
+use crate::parquet::Scenario;
+
+use super::make_test_file_rg;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_file_one_column(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![make_int64_batches_with_null(
+ num_null,
+ no_null_values_start,
+ no_null_values_end,
+ )];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ // close file
+ let _file_meta = writer.close().unwrap();
+
+ // open the file & get the reader
+ let file = output_file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+// Create a parquet file with many columns each has different data type
+// - Data types are specified by the given scenario
+// - Row group sizes are withe the same or different depending on the provided row_per_group & data created in the scenario
+pub async fn parquet_file_many_columns(
+ scenario: super::Scenario,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let file = make_test_file_rg(scenario, row_per_group).await;
+
+ // open the file & get the reader
+ let file = file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+struct Test {
+ reader: ParquetRecordBatchReaderBuilder<File>,
+ expected_min: ArrayRef,
+ expected_max: ArrayRef,
+ expected_null_counts: UInt64Array,
+ expected_row_counts: UInt64Array,
+}
+
+impl Test {
+ fn run(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min,
+ expected_max,
+ expected_null_counts,
+ expected_row_counts,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+
+ assert_eq!(&min, &expected_min, "Mismatch with expected minimums");
+
+ let max = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Max,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ assert_eq!(&max, &expected_max, "Mismatch with expected maximum");
+
+ let null_counts = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::NullCount,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ let expected_null_counts = Arc::new(expected_null_counts) as ArrayRef;
+ assert_eq!(
+ &null_counts, &expected_null_counts,
+ "Mismatch with expected null counts"
+ );
+
+ let row_counts = StatisticsConverter::row_counts(reader.metadata()).unwrap();
+ assert_eq!(
+ row_counts, expected_row_counts,
+ "Mismatch with expected row counts"
+ );
+ }
+
+ fn run_col_not_found(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min: _,
+ expected_max: _,
+ expected_null_counts: _,
+ expected_row_counts: _,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ );
+
+ assert!(min.is_err());
+ }
+}
+
+// TESTS
+//
+// Remaining cases
+// - Create parquet files / metadata with missing statistic values
+// - Create parquet files / metadata with different data types -- included but not all data types yet
+// - Create parquet files / metadata with different row group sizes -- done
+// - Using truncated statistics ("exact min value" and "exact max value" https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.max_is_exact)
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(0, 4, 7, row_per_group);
+ Test {
+ reader,
+ // min is 4
+ expected_min: Arc::new(Int64Array::from(vec![4])),
+ // max is 6
+ expected_max: Arc::new(Int64Array::from(vec![6])),
+ // no nulls
+ expected_null_counts: UInt64Array::from(vec![0]),
+ // 3 rows
+ expected_row_counts: UInt64Array::from(vec![3]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_one_row_group_with_null_and_negative() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(2, -1, 5, row_per_group);
+
+ Test {
+ reader,
+ // min is -1
+ expected_min: Arc::new(Int64Array::from(vec![-1])),
+ // max is 4
+ expected_max: Arc::new(Int64Array::from(vec![4])),
+ // 2 nulls
+ expected_null_counts: UInt64Array::from(vec![2]),
+ // 8 rows
+ expected_row_counts: UInt64Array::from(vec![8]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_group_with_null() {
+ let row_per_group = 10;
+ let reader = parquet_file_one_column(2, 4, 17, row_per_group);
+
+ Test {
+ reader,
+ // mins are [4, 14]
+ expected_min: Arc::new(Int64Array::from(vec![4, 14])),
+ // maxes are [13, 16]
+ expected_max: Arc::new(Int64Array::from(vec![13, 16])),
+ // nulls are [0, 2]
+ expected_null_counts: UInt64Array::from(vec![0, 2]),
+ // row counts are [10, 5]
+ expected_row_counts: UInt64Array::from(vec![10, 5]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_groups_with_all_nulls_in_one() {
+ let row_per_group = 5;
+ let reader = parquet_file_one_column(4, -2, 2, row_per_group);
+
+ Test {
+ reader,
+ // mins are [-2, null]
+ expected_min: Arc::new(Int64Array::from(vec![Some(-2), None])),
+ // maxes are [1, null]
+ expected_max: Arc::new(Int64Array::from(vec![Some(1), None])),
+ // nulls are [1, 3]
+ expected_null_counts: UInt64Array::from(vec![1, 3]),
+ // row counts are [5, 3]
+ expected_row_counts: UInt64Array::from(vec![5, 3]),
+ }
+ .run("i64")
+}
+
+/////////////// MORE GENERAL TESTS //////////////////////
+// . Many columns in a file
+// . Differnet data types
+// . Different row group sizes
+
+// Four different integer types
+#[tokio::test]
+async fn test_int_64() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i64");
+}
+
+#[tokio::test]
+async fn test_int_32() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int32Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int32Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i32");
+}
+
+// BUG: ignore this test for now
+// Note that the file has 4 columns named "i8", "i16", "i32", "i64".
+// - The tests on column i32 and i64 passed.
+// - The tests on column i8 and i16 failed.
+#[ignore]
+#[tokio::test]
+async fn test_int_16() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array type even though I debugged and the columns name is "i16" an its data is Int16
+ // My debugging tells me the bug is either at:
+ // 1. The new code to get "iter". See the code in this PR with
+ // // Get an iterator over the column statistics
+ // let iter = row_groups
+ // .iter()
+ // .map(|x| x.column(parquet_idx).statistics());
+ // OR
+ // 2. in the function (and/or its marco) `pub(crate) fn min_statistics<'a, I: Iterator<Item = Option<&'a ParquetStatistics>>>` here
+ // https://github.com/apache/datafusion/blob/ea023e2d4878240eece870cf4b346c7a0667aeed/datafusion/core/src/datasource/physical_plan/parquet/statistics.rs#L179
+ expected_min: Arc::new(Int16Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int16Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i16");
+}
+
+// BUG (same as above): ignore this test for now
+#[ignore]
+#[tokio::test]
+async fn test_int_8() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ // BUG: not sure why this returns same data but in Int32Array even though I debugged and the columns name is "i8" an its data is Int8
+ expected_min: Arc::new(Int8Array::from(vec![-5, -4, 0, 5])), // panic here because the actual data is Int32Array
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int8Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i8");
+}
+
+// timestamp
+#[tokio::test]
+async fn test_timestamp() {
+ let row_per_group = 5;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches, each has 5 rowws.
+ // Since the row group isze is set to 5, those 4 batches will go into 4 row groups
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577840471000000000, 1577841061000000000, 1578704461000000000,]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577840471000000000,
+ 1577841061000000000,
+ 1578704461000000000,
+ ])),
+ // maxes are [1577926861000000000, 1577926871000000000, 1577927461000000000, 1578790861000000000,]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1577926871000000000,
+ 1577927461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 1, 1, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577840471000000,
+ 1577841061000000,
+ 1578704461000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1577926871000000,
+ 1577927461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577840471000,
+ 1577841061000,
+ 1578704461000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1577926871000,
+ 1577927461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577840471, 1577841061, 1578704461,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1577926871, 1577927461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 1, 1, 1]),
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("seconds");
+}
+
+// timestamp with different row group sizes
+#[tokio::test]
+async fn test_timestamp_diff_rg_sizes() {
+ let row_per_group = 8;
+
+ // This creates a parquet files of 5 columns named "nanos", "micros", "millis", "seconds", "names"
+ // "nanos" --> TimestampNanosecondArray
+ // "micros" --> TimestampMicrosecondArray
+ // "millis" --> TimestampMillisecondArray
+ // "seconds" --> TimestampSecondArray
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 3 row groups with size 8, 8, 4
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [1577840461000000000, 1577841061000000000, 1578704521000000000]
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000000,
+ 1577841061000000000,
+ 1578704521000000000,
+ ])),
+ // maxes are [1577926861000000000, 1578704461000000000, 157879086100000000]
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000000,
+ 1578704461000000000,
+ 1578790861000000000,
+ ])),
+ // nulls are [1, 2, 1]
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ // row counts are [8, 8, 4]
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("nanos");
+
+ // micros
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000000,
+ 1577841061000000,
+ 1578704521000000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000000,
+ 1578704461000000,
+ 1578790861000000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("micros");
+
+ // millis
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461000,
+ 1577841061000,
+ 1578704521000,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861000,
+ 1578704461000,
+ 1578790861000,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("millis");
+
+ // seconds
+ let reader = parquet_file_many_columns(Scenario::Timestamps, row_per_group).await;
+ Test {
+ reader,
+ expected_min: Arc::new(Int64Array::from(vec![
+ 1577840461, 1577841061, 1578704521,
+ ])),
+ expected_max: Arc::new(Int64Array::from(vec![
+ 1577926861, 1578704461, 1578790861,
+ ])),
+ expected_null_counts: UInt64Array::from(vec![1, 2, 1]),
+ expected_row_counts: UInt64Array::from(vec![8, 8, 4]),
+ }
+ .run("seconds");
+}
+
+// date with different row group sizes
+#[tokio::test]
+async fn test_dates_32_diff_rg_sizes() {
+ let row_per_group = 13;
+
+ // This creates a parquet files of 3 columns named "date32", "date64", "names"
+ // "date32" --> Date32Array
+ // "date64" --> Date64Array
+ // "names" --> StringArray
+ //
+ // The file is created by 4 record batches (each has a null row), each has 5 rows but then will be split into 2 row groups with size 13, 7
+ let reader = parquet_file_many_columns(Scenario::Dates, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [18262, 18565,]
+ expected_min: Arc::new(Int32Array::from(vec![18262, 18565])),
Review Comment:
https://github.com/apache/datafusion/issues/10587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: some tests to write data to a parquet file and read its metadata [datafusion]
Posted by "NGA-TRAN (via GitHub)" <gi...@apache.org>.
NGA-TRAN commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1603501230
##########
datafusion/core/src/datasource/physical_plan/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,43 @@
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+use datafusion_common::Result;
+use parquet::file::statistics::Statistics as ParquetStatistics;
+
+/// statistics extracted from `Statistics` as Arrow `ArrayRef`s
+///
+/// # Note:
+/// If the corresponding `Statistics` is not present, or has no information for
+/// a column, a NULL is present in the corresponding array entry
+pub struct ArrowStatistics {
+ /// min values
+ min: ArrayRef,
+ /// max values
+ max: ArrayRef,
+ /// Row counts (UInt64Array)
+ row_count: ArrayRef,
+ /// Null Counts (UInt64Array)
+ null_count: ArrayRef,
+}
+
+/// Extract `ArrowStatistics` from the parquet [`Statistics`]
+pub fn parquet_stats_to_arrow<'a>(
+ arrow_datatype: &DataType,
+ statistics: impl IntoIterator<Item = Option<&'a ParquetStatistics>>,
+) -> Result<ArrowStatistics> {
+ todo!() // MY TODO next
+}
Review Comment:
Yeah, it is very easy to get `FileMetaData` from the parquet reader. I agree the sort order is not needed (yet) but I will see what we needs we we go and add them in
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,528 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::sync::Arc;
+
+// use arrow::json::reader;
+use arrow_array::{make_array, Array, Int64Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+use parquet::file::statistics::Statistics as ParquetFileStatistics;
+use parquet::format::Statistics as ParquetFormatStatistics;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// TODO: Eventually, we will need only read_statistics and read_row_count,
+// but for my understading, we will keep both write and read statistics.
+// Their values should be the same
+pub struct RowGroupsStatistics {
+ // Statistics from the file writer
+ pub write_statistics: Vec<WriteRowGroupStatistics>,
+
+ // Statistics from the file reader
+ pub read_statistics: Vec<ReadRowGroupStatistics>,
+}
+
+pub struct WriteRowGroupStatistics {
+ pub statistics: ParquetFormatStatistics,
+ pub row_count: i64,
+}
+
+pub struct ReadRowGroupStatistics {
+ pub statistics: ParquetFileStatistics,
+ pub row_count: i64,
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_row_group_statistics(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> RowGroupsStatistics {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![
+ make_int64_batches_with_null(num_null, no_null_values_start, no_null_values_end), // TODO: likely make this more general using
+ // create_data_batch(scenario); where scenario is the respective enum data type
+ ];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ //////////////// WRITE STATISTICS ///////////////////////
+ let file_meta = writer.close().unwrap();
+
+ // meta data of the row groups
+ let rg_metas = file_meta.row_groups;
+
+ // Statstics of each grow group
+ let mut write_rg_statistics = Vec::new();
+ for rg_meta in rg_metas {
+ let column = &rg_meta.columns;
+ assert_eq!(column.len(), 1, "expected 1 column");
+ let col_metadata = &column[0].meta_data;
+ let col_metadata = col_metadata.as_ref().unwrap();
+ let stats = &col_metadata.statistics;
+ let stats = stats.as_ref().unwrap();
+
+ let rg_stats = WriteRowGroupStatistics {
+ row_count: rg_meta.num_rows,
+ statistics: stats.clone(),
+ };
+ write_rg_statistics.push(rg_stats);
+ }
+
+ //////////////// READ STATISTICS ///////////////////////
+ let file = output_file.reopen().unwrap();
+
+ let reader = ArrowReaderBuilder::try_new(file).unwrap();
+ let parquet_meta = reader.metadata().clone();
+
+ // meta data of the row groups
+ let rg_metas = parquet_meta.row_groups();
+
+ // Statstics of each grow group
+ let mut read_rg_statistics = Vec::new();
+ for rg_meta in rg_metas {
+ let column = &rg_meta.columns();
+ assert_eq!(column.len(), 1, "expected 1 column");
+ let stats = &column[0].statistics().unwrap();
+
+ let rg_stats = ReadRowGroupStatistics {
+ row_count: rg_meta.num_rows() as i64,
+ statistics: (*stats).clone(),
+ };
+ read_rg_statistics.push(rg_stats);
+ }
+
+ RowGroupsStatistics {
+ write_statistics: write_rg_statistics,
+ read_statistics: read_rg_statistics,
+ }
+}
+
+// TESTS
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let rg_statistics = parquet_row_group_statistics(0, 4, 7, row_per_group);
+
+ //////////// Verify write row group statistics ////////////
+ let write_rg_stats = &rg_statistics.write_statistics;
+
+ // only 1 row group
+ assert_eq!(write_rg_stats.len(), 1, "expected 1 row group");
+ let rg = &write_rg_stats[0];
+
+ // 3 rows
+ assert_eq!(rg.row_count, 3, "expected 3 rows");
+
+ // no nulls
+ assert!(rg.statistics.null_count.is_none(), "expected no nulls");
+
+ let statistics = &rg.statistics;
+
+ // min is [4, 0, 0, 0, 0, 0, 0, 0]
+ assert_eq!(
+ *statistics.min_value.as_ref().unwrap(),
+ [4, 0, 0, 0, 0, 0, 0, 0]
+ );
+ // max is [6, 0, 0, 0, 0, 0, 0, 0]
+ assert_eq!(
+ *statistics.max_value.as_ref().unwrap(),
+ [6, 0, 0, 0, 0, 0, 0, 0]
+ );
+
+ println!("RG statistics:\n {statistics:#?}");
+ // This will display:
+ // RG statistics:
+ // Statistics {
+ // max: Some(
+ // [
+ // 6,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // ],
+ // ),
+ // min: Some(
+ // [
+ // 4,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // ],
+ // ),
+ // null_count: None,
+ // distinct_count: None,
+ // max_value: Some(
+ // [
+ // 6,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // ],
+ // ),
+ // min_value: Some(
+ // [
+ // 4,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // ],
+ // ),
+ // is_max_value_exact: Some(
+ // true,
+ // ),
+ // is_min_value_exact: Some(
+ // true,
+ // ),
+ // }
+
+ //////////// Verify read row group statistics ////////////
+ let read_rg_stats = &rg_statistics.read_statistics;
+
+ // only 1 row group
+ assert_eq!(read_rg_stats.len(), 1, "expected 1 row group");
+ let rg = &read_rg_stats[0];
+
+ // 3 rows
+ assert_eq!(rg.row_count, 3, "expected 3 rows");
+
+ // rg.statistics is Int64
+ let stats = &rg.statistics;
+ match stats {
+ ParquetFileStatistics::Int64(int64_stats) => {
+ // no nulls
+ assert_eq!(int64_stats.null_count(), 0, "expected no nulls");
+ // min is 4
+ assert_eq!(*int64_stats.min(), 4);
+ // max is 6
+ assert_eq!(*int64_stats.max(), 6);
+ }
+ _ => panic!("expected Int64"),
+ }
Review Comment:
Yes, that is the plan. These tests are mostly for me to understand what the parquet stats looks like before I convert it into the arrow stats
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: some tests to write data to a parquet file and read its metadata [datafusion]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1603013478
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,528 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::sync::Arc;
+
+// use arrow::json::reader;
+use arrow_array::{make_array, Array, Int64Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+use parquet::file::statistics::Statistics as ParquetFileStatistics;
+use parquet::format::Statistics as ParquetFormatStatistics;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// TODO: Eventually, we will need only read_statistics and read_row_count,
+// but for my understading, we will keep both write and read statistics.
+// Their values should be the same
+pub struct RowGroupsStatistics {
+ // Statistics from the file writer
+ pub write_statistics: Vec<WriteRowGroupStatistics>,
+
+ // Statistics from the file reader
+ pub read_statistics: Vec<ReadRowGroupStatistics>,
+}
+
+pub struct WriteRowGroupStatistics {
+ pub statistics: ParquetFormatStatistics,
+ pub row_count: i64,
+}
+
+pub struct ReadRowGroupStatistics {
+ pub statistics: ParquetFileStatistics,
+ pub row_count: i64,
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_row_group_statistics(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> RowGroupsStatistics {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![
+ make_int64_batches_with_null(num_null, no_null_values_start, no_null_values_end), // TODO: likely make this more general using
+ // create_data_batch(scenario); where scenario is the respective enum data type
+ ];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ //////////////// WRITE STATISTICS ///////////////////////
+ let file_meta = writer.close().unwrap();
Review Comment:
One is the native parquet encoding, one is the Rust version of it. In general you should avoid using the `format::statistics` directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: some tests to write data to a parquet file and read its metadata [datafusion]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1603007745
##########
datafusion/core/src/datasource/physical_plan/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,43 @@
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+use datafusion_common::Result;
+use parquet::file::statistics::Statistics as ParquetStatistics;
+
+/// statistics extracted from `Statistics` as Arrow `ArrayRef`s
+///
+/// # Note:
+/// If the corresponding `Statistics` is not present, or has no information for
+/// a column, a NULL is present in the corresponding array entry
+pub struct ArrowStatistics {
+ /// min values
+ min: ArrayRef,
+ /// max values
+ max: ArrayRef,
+ /// Row counts (UInt64Array)
+ row_count: ArrayRef,
+ /// Null Counts (UInt64Array)
+ null_count: ArrayRef,
+}
+
+/// Extract `ArrowStatistics` from the parquet [`Statistics`]
+pub fn parquet_stats_to_arrow<'a>(
+ arrow_datatype: &DataType,
+ statistics: impl IntoIterator<Item = Option<&'a ParquetStatistics>>,
+) -> Result<ArrowStatistics> {
+ todo!() // MY TODO next
+}
Review Comment:
To emphasise the point I made when this API was originally proposed, you need more than just the ParquetStatistics in order to correctly interpret the data. You need at least the FileMetadata to get the https://docs.rs/parquet/latest/parquet/file/metadata/struct.FileMetaData.html#method.column_order in order to be able to even interpret what the statistics mean for a given column
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: some tests to write data to a parquet file and read its metadata [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1603880876
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,528 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::sync::Arc;
+
+// use arrow::json::reader;
+use arrow_array::{make_array, Array, Int64Array, RecordBatch};
+use arrow_schema::{DataType, Field, Schema};
+use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+use parquet::file::statistics::Statistics as ParquetFileStatistics;
+use parquet::format::Statistics as ParquetFormatStatistics;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// TODO: Eventually, we will need only read_statistics and read_row_count,
+// but for my understading, we will keep both write and read statistics.
+// Their values should be the same
+pub struct RowGroupsStatistics {
+ // Statistics from the file writer
+ pub write_statistics: Vec<WriteRowGroupStatistics>,
+
+ // Statistics from the file reader
+ pub read_statistics: Vec<ReadRowGroupStatistics>,
+}
+
+pub struct WriteRowGroupStatistics {
+ pub statistics: ParquetFormatStatistics,
+ pub row_count: i64,
+}
+
+pub struct ReadRowGroupStatistics {
+ pub statistics: ParquetFileStatistics,
+ pub row_count: i64,
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_row_group_statistics(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> RowGroupsStatistics {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![
+ make_int64_batches_with_null(num_null, no_null_values_start, no_null_values_end), // TODO: likely make this more general using
+ // create_data_batch(scenario); where scenario is the respective enum data type
+ ];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ //////////////// WRITE STATISTICS ///////////////////////
+ let file_meta = writer.close().unwrap();
+
+ // meta data of the row groups
+ let rg_metas = file_meta.row_groups;
+
+ // Statstics of each grow group
+ let mut write_rg_statistics = Vec::new();
+ for rg_meta in rg_metas {
+ let column = &rg_meta.columns;
+ assert_eq!(column.len(), 1, "expected 1 column");
+ let col_metadata = &column[0].meta_data;
+ let col_metadata = col_metadata.as_ref().unwrap();
+ let stats = &col_metadata.statistics;
+ let stats = stats.as_ref().unwrap();
+
+ let rg_stats = WriteRowGroupStatistics {
+ row_count: rg_meta.num_rows,
+ statistics: stats.clone(),
+ };
+ write_rg_statistics.push(rg_stats);
+ }
+
+ //////////////// READ STATISTICS ///////////////////////
+ let file = output_file.reopen().unwrap();
+
+ let reader = ArrowReaderBuilder::try_new(file).unwrap();
+ let parquet_meta = reader.metadata().clone();
+
+ // meta data of the row groups
+ let rg_metas = parquet_meta.row_groups();
+
+ // Statstics of each grow group
+ let mut read_rg_statistics = Vec::new();
+ for rg_meta in rg_metas {
+ let column = &rg_meta.columns();
+ assert_eq!(column.len(), 1, "expected 1 column");
+ let stats = &column[0].statistics().unwrap();
+
+ let rg_stats = ReadRowGroupStatistics {
+ row_count: rg_meta.num_rows() as i64,
+ statistics: (*stats).clone(),
+ };
+ read_rg_statistics.push(rg_stats);
+ }
+
+ RowGroupsStatistics {
+ write_statistics: write_rg_statistics,
+ read_statistics: read_rg_statistics,
+ }
+}
+
+// TESTS
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let rg_statistics = parquet_row_group_statistics(0, 4, 7, row_per_group);
+
+ //////////// Verify write row group statistics ////////////
+ let write_rg_stats = &rg_statistics.write_statistics;
+
+ // only 1 row group
+ assert_eq!(write_rg_stats.len(), 1, "expected 1 row group");
+ let rg = &write_rg_stats[0];
+
+ // 3 rows
+ assert_eq!(rg.row_count, 3, "expected 3 rows");
+
+ // no nulls
+ assert!(rg.statistics.null_count.is_none(), "expected no nulls");
+
+ let statistics = &rg.statistics;
+
+ // min is [4, 0, 0, 0, 0, 0, 0, 0]
+ assert_eq!(
+ *statistics.min_value.as_ref().unwrap(),
+ [4, 0, 0, 0, 0, 0, 0, 0]
+ );
+ // max is [6, 0, 0, 0, 0, 0, 0, 0]
+ assert_eq!(
+ *statistics.max_value.as_ref().unwrap(),
+ [6, 0, 0, 0, 0, 0, 0, 0]
+ );
+
+ println!("RG statistics:\n {statistics:#?}");
+ // This will display:
+ // RG statistics:
+ // Statistics {
+ // max: Some(
+ // [
+ // 6,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // ],
+ // ),
+ // min: Some(
+ // [
+ // 4,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // ],
+ // ),
+ // null_count: None,
+ // distinct_count: None,
+ // max_value: Some(
+ // [
+ // 6,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // ],
+ // ),
+ // min_value: Some(
+ // [
+ // 4,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // 0,
+ // ],
+ // ),
+ // is_max_value_exact: Some(
+ // true,
+ // ),
+ // is_min_value_exact: Some(
+ // true,
+ // ),
+ // }
+
+ //////////// Verify read row group statistics ////////////
+ let read_rg_stats = &rg_statistics.read_statistics;
+
+ // only 1 row group
+ assert_eq!(read_rg_stats.len(), 1, "expected 1 row group");
+ let rg = &read_rg_stats[0];
+
+ // 3 rows
+ assert_eq!(rg.row_count, 3, "expected 3 rows");
+
+ // rg.statistics is Int64
+ let stats = &rg.statistics;
+ match stats {
+ ParquetFileStatistics::Int64(int64_stats) => {
+ // no nulls
+ assert_eq!(int64_stats.null_count(), 0, "expected no nulls");
+ // min is 4
+ assert_eq!(*int64_stats.min(), 4);
+ // max is 6
+ assert_eq!(*int64_stats.max(), 6);
+ }
+ _ => panic!("expected Int64"),
+ }
Review Comment:
Makes sense -- sorry this wasn't as ready to work on as I may have hoped
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: API for collecting statistics/index for metadata of a parquet file [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1607087345
##########
datafusion/core/tests/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,652 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test of extracting statitics from parquet files.
+//! It writes data into a parquet file, reads statistics and verifies they are correct
+
+use std::fs::File;
+use std::sync::Arc;
+
+use arrow_array::{
+ make_array, Array, ArrayRef, Int16Array, Int32Array, Int64Array, Int8Array,
+ RecordBatch, UInt64Array,
+};
+use arrow_schema::{DataType, Field, Schema};
+use datafusion::datasource::physical_plan::parquet::{
+ RequestedStatistics, StatisticsConverter,
+};
+use parquet::arrow::arrow_reader::{ArrowReaderBuilder, ParquetRecordBatchReaderBuilder};
+use parquet::arrow::ArrowWriter;
+use parquet::file::properties::{EnabledStatistics, WriterProperties};
+
+use crate::parquet::Scenario;
+
+use super::make_test_file_rg;
+
+// TEST HELPERS
+
+/// Return a record batch with i64 with Null values
+fn make_int64_batches_with_null(
+ null_values: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("i64", DataType::Int64, true)]));
+
+ let v64: Vec<i64> = (no_null_values_start as _..no_null_values_end as _).collect();
+
+ RecordBatch::try_new(
+ schema,
+ vec![make_array(
+ Int64Array::from_iter(
+ v64.into_iter()
+ .map(Some)
+ .chain(std::iter::repeat(None).take(null_values)),
+ )
+ .to_data(),
+ )],
+ )
+ .unwrap()
+}
+
+// Create a parquet file with one column for data type i64
+// Data of the file include
+// . Number of null rows is the given num_null
+// . There are non-null values in the range [no_null_values_start, no_null_values_end], one value each row
+// . The file is divided into row groups of size row_per_group
+pub fn parquet_file_one_column(
+ num_null: usize,
+ no_null_values_start: i64,
+ no_null_values_end: i64,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let mut output_file = tempfile::Builder::new()
+ .prefix("parquert_statistics_test")
+ .suffix(".parquet")
+ .tempfile()
+ .expect("tempfile creation");
+
+ let props = WriterProperties::builder()
+ .set_max_row_group_size(row_per_group)
+ .set_statistics_enabled(EnabledStatistics::Chunk)
+ .build();
+
+ let batches = vec![make_int64_batches_with_null(
+ num_null,
+ no_null_values_start,
+ no_null_values_end,
+ )];
+
+ let schema = batches[0].schema();
+
+ let mut writer = ArrowWriter::try_new(&mut output_file, schema, Some(props)).unwrap();
+
+ for batch in batches {
+ writer.write(&batch).expect("writing batch");
+ }
+
+ // close file
+ let _file_meta = writer.close().unwrap();
+
+ // open the file & get the reader
+ let file = output_file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+// Create a parquet file with many columns each has different data type
+// - Data types are specified by the given scenario
+// - Row group sizes are withe the same or different depending on the provided row_per_group & data created in the scenario
+pub async fn parquet_file_many_columns(
+ scenario: super::Scenario,
+ row_per_group: usize,
+) -> ParquetRecordBatchReaderBuilder<File> {
+ let file = make_test_file_rg(scenario, row_per_group).await;
+
+ // open the file & get the reader
+ let file = file.reopen().unwrap();
+ ArrowReaderBuilder::try_new(file).unwrap()
+}
+
+struct Test {
+ reader: ParquetRecordBatchReaderBuilder<File>,
+ expected_min: ArrayRef,
+ expected_max: ArrayRef,
+ expected_null_counts: UInt64Array,
+ expected_row_counts: UInt64Array,
+}
+
+impl Test {
+ fn run(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min,
+ expected_max,
+ expected_null_counts,
+ expected_row_counts,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+
+ assert_eq!(&min, &expected_min, "Mismatch with expected minimums");
+
+ let max = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Max,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ assert_eq!(&max, &expected_max, "Mismatch with expected maximum");
+
+ let null_counts = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::NullCount,
+ reader.schema(),
+ )
+ .unwrap()
+ .extract(reader.metadata())
+ .unwrap();
+ let expected_null_counts = Arc::new(expected_null_counts) as ArrayRef;
+ assert_eq!(
+ &null_counts, &expected_null_counts,
+ "Mismatch with expected null counts"
+ );
+
+ let row_counts = StatisticsConverter::row_counts(reader.metadata()).unwrap();
+ assert_eq!(
+ row_counts, expected_row_counts,
+ "Mismatch with expected row counts"
+ );
+ }
+
+ fn run_col_not_found(self, col_name: &str) {
+ let Self {
+ reader,
+ expected_min: _,
+ expected_max: _,
+ expected_null_counts: _,
+ expected_row_counts: _,
+ } = self;
+
+ let min = StatisticsConverter::try_new(
+ col_name,
+ RequestedStatistics::Min,
+ reader.schema(),
+ );
+
+ assert!(min.is_err());
+ }
+}
+
+// TESTS
+//
+// Remaining cases
+// - Create parquet files / metadata with missing statistic values
+// - Create parquet files / metadata with different data types -- included but not all data types yet
+// - Create parquet files / metadata with different row group sizes -- done
+// - Using truncated statistics ("exact min value" and "exact max value" https://docs.rs/parquet/latest/parquet/file/statistics/enum.Statistics.html#method.max_is_exact)
+
+#[tokio::test]
+async fn test_one_row_group_without_null() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(0, 4, 7, row_per_group);
+ Test {
+ reader,
+ // min is 4
+ expected_min: Arc::new(Int64Array::from(vec![4])),
+ // max is 6
+ expected_max: Arc::new(Int64Array::from(vec![6])),
+ // no nulls
+ expected_null_counts: UInt64Array::from(vec![0]),
+ // 3 rows
+ expected_row_counts: UInt64Array::from(vec![3]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_one_row_group_with_null_and_negative() {
+ let row_per_group = 20;
+ let reader = parquet_file_one_column(2, -1, 5, row_per_group);
+
+ Test {
+ reader,
+ // min is -1
+ expected_min: Arc::new(Int64Array::from(vec![-1])),
+ // max is 4
+ expected_max: Arc::new(Int64Array::from(vec![4])),
+ // 2 nulls
+ expected_null_counts: UInt64Array::from(vec![2]),
+ // 8 rows
+ expected_row_counts: UInt64Array::from(vec![8]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_group_with_null() {
+ let row_per_group = 10;
+ let reader = parquet_file_one_column(2, 4, 17, row_per_group);
+
+ Test {
+ reader,
+ // mins are [4, 14]
+ expected_min: Arc::new(Int64Array::from(vec![4, 14])),
+ // maxes are [13, 16]
+ expected_max: Arc::new(Int64Array::from(vec![13, 16])),
+ // nulls are [0, 2]
+ expected_null_counts: UInt64Array::from(vec![0, 2]),
+ // row counts are [10, 5]
+ expected_row_counts: UInt64Array::from(vec![10, 5]),
+ }
+ .run("i64")
+}
+
+#[tokio::test]
+async fn test_two_row_groups_with_all_nulls_in_one() {
+ let row_per_group = 5;
+ let reader = parquet_file_one_column(4, -2, 2, row_per_group);
+
+ Test {
+ reader,
+ // mins are [-2, null]
+ expected_min: Arc::new(Int64Array::from(vec![Some(-2), None])),
+ // maxes are [1, null]
+ expected_max: Arc::new(Int64Array::from(vec![Some(1), None])),
+ // nulls are [1, 3]
+ expected_null_counts: UInt64Array::from(vec![1, 3]),
+ // row counts are [5, 3]
+ expected_row_counts: UInt64Array::from(vec![5, 3]),
+ }
+ .run("i64")
+}
+
+/////////////// MORE GENERAL TESTS //////////////////////
+// . Many columns in a file
+// . Differnet data types
+// . Different row group sizes
+
+// Four different integer types
+#[tokio::test]
+async fn test_int_64() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int64Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int64Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i64");
+}
+
+#[tokio::test]
+async fn test_int_32() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
+ expected_min: Arc::new(Int32Array::from(vec![-5, -4, 0, 5])),
+ // maxes are [-1, 0, 4, 9]
+ expected_max: Arc::new(Int32Array::from(vec![-1, 0, 4, 9])),
+ // nulls are [0, 0, 0, 0]
+ expected_null_counts: UInt64Array::from(vec![0, 0, 0, 0]),
+ // row counts are [5, 5, 5, 5]
+ expected_row_counts: UInt64Array::from(vec![5, 5, 5, 5]),
+ }
+ .run("i32");
+}
+
+// BUG: ignore this test for now
+// Note that the file has 4 columns named "i8", "i16", "i32", "i64".
+// - The tests on column i32 and i64 passed.
+// - The tests on column i8 and i16 failed.
+#[ignore]
+#[tokio::test]
+async fn test_int_16() {
+ let row_per_group = 5;
+ // This creates a parquet files of 4 columns named "i8", "i16", "i32", "i64"
+ let reader = parquet_file_many_columns(Scenario::Int, row_per_group).await;
+
+ Test {
+ reader,
+ // mins are [-5, -4, 0, 5]
Review Comment:
This may be related to https://github.com/apache/datafusion/issues/9779 (something related to how parquet handles signed integers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: some tests to write data to a parquet file and read its metadata [datafusion]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1603007745
##########
datafusion/core/src/datasource/physical_plan/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,43 @@
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+use datafusion_common::Result;
+use parquet::file::statistics::Statistics as ParquetStatistics;
+
+/// statistics extracted from `Statistics` as Arrow `ArrayRef`s
+///
+/// # Note:
+/// If the corresponding `Statistics` is not present, or has no information for
+/// a column, a NULL is present in the corresponding array entry
+pub struct ArrowStatistics {
+ /// min values
+ min: ArrayRef,
+ /// max values
+ max: ArrayRef,
+ /// Row counts (UInt64Array)
+ row_count: ArrayRef,
+ /// Null Counts (UInt64Array)
+ null_count: ArrayRef,
+}
+
+/// Extract `ArrowStatistics` from the parquet [`Statistics`]
+pub fn parquet_stats_to_arrow<'a>(
+ arrow_datatype: &DataType,
+ statistics: impl IntoIterator<Item = Option<&'a ParquetStatistics>>,
+) -> Result<ArrowStatistics> {
+ todo!() // MY TODO next
+}
Review Comment:
To emphasise the point I made when this API was originally proposed, you need more than just the ParquetStatistics in order to correctly interpret the data. You need at least the FileMetadata to get the https://docs.rs/parquet/latest/parquet/file/metadata/struct.FileMetaData.html#method.column_order in order to be able to even interpret what the statistics mean for a given column.
Additionally you need to actually have the parquet schema as the **arrow datatype may not match what the parquet data is encoded as**. The parquet schema is authoritative when reading parquet data, the arrow datatype is purely what the data should be coerced to once read from parquet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: some tests to write data to a parquet file and read its metadata [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#discussion_r1603344193
##########
datafusion/core/src/datasource/physical_plan/parquet/arrow_statistics.rs:
##########
@@ -0,0 +1,43 @@
+use arrow_array::ArrayRef;
+use arrow_schema::DataType;
+use datafusion_common::Result;
+use parquet::file::statistics::Statistics as ParquetStatistics;
+
+/// statistics extracted from `Statistics` as Arrow `ArrayRef`s
+///
+/// # Note:
+/// If the corresponding `Statistics` is not present, or has no information for
+/// a column, a NULL is present in the corresponding array entry
+pub struct ArrowStatistics {
+ /// min values
+ min: ArrayRef,
+ /// max values
+ max: ArrayRef,
+ /// Row counts (UInt64Array)
+ row_count: ArrayRef,
+ /// Null Counts (UInt64Array)
+ null_count: ArrayRef,
+}
+
+/// Extract `ArrowStatistics` from the parquet [`Statistics`]
+pub fn parquet_stats_to_arrow<'a>(
+ arrow_datatype: &DataType,
+ statistics: impl IntoIterator<Item = Option<&'a ParquetStatistics>>,
+) -> Result<ArrowStatistics> {
+ todo!() // MY TODO next
+}
Review Comment:
In terms of "column order" I think we initially should do what DataFusion currently does with ColumnOrder (which is ignore it) and file a ticket to handle it longer term
Including the parquet schema is a good idea. I think this will become more obvious as we begin writing these tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: API for collecting statistics/index for metadata of a parquet file + tests [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10537:
URL: https://github.com/apache/datafusion/pull/10537#issuecomment-2120986025
I have filed the following tickets
* https://github.com/apache/datafusion/issues/10585
* https://github.com/apache/datafusion/issues/10586
* #10587
I think this PR is now ready to go. I plan to merge it in when the CI passes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org