You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "diegoQuinas (via GitHub)" <gi...@apache.org> on 2026/09/23 00:41:05 UTC
[PR] feat: expose bloom filter, page index and sorting metadata in parquet… [datafusion]
diegoQuinas opened a new pull request, #25638:
URL: https://github.com/apache/datafusion/pull/25638
…_metadata
Add footer fields already present on ColumnChunkMetaData and RowGroupMetaData to the datafusion-cli parquet_metadata table function.
## Which issue does this PR close?
This PR closes one of the three items of the issue #25499
## What changes are included in this PR?
It changes datafusion-cli main and functions file, also it updates the docs for the new columns
## What is the testing strategy for this PR?
This feature is covered by the unit tests and snapshot tests of datafusion-cli
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'.
If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it.
You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised.
-->
## Are there any user-facing changes?
Yes, there are user-facing changes that require documentation (also included in this PR)
No, no breaking changes to public APIs, (beware that the datafusion-cli will exponse more fields in the output)
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: expose bloom filter, page index and sorting metadata in parquet… [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25638:
URL: https://github.com/apache/datafusion/pull/25638#issuecomment-6406204707
Marked as draft after 14 days without activity: merge conflicts.
Please address these items and mark **Ready for review** when ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: expose bloom filter, page index and sorting metadata in parquet… [datafusion]
Posted by "rgbuilds (via GitHub)" <gi...@apache.org>.
rgbuilds commented on code in PR #25638:
URL: https://github.com/apache/datafusion/pull/25638#discussion_r4090493569
##########
datafusion-cli/src/main.rs:
##########
@@ -599,11 +599,11 @@ mod tests {
let rbs = df.collect().await?;
assert_snapshot!(batches_to_string(&rbs),@r#"
- +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+
- | filename | row_group_id | row_group_num_rows | row_group_num_columns | row_group_bytes | column_id | file_offset | num_values | path_in_schema | type | stats_min | stats_max | stats_null_count | stats_distinct_count | stats_min_value | stats_max_value | compression | encodings | index_page_offset | dictionary_page_offset | data_page_offset | total_compressed_size | total_uncompressed_size |
- +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+
- | ../parquet-testing/data/data_index_bloom_encoding_stats.parquet | 0 | 14 | 1 | 163 | 0 | 4 | 14 | "String" | BYTE_ARRAY | Hello | today | 0 | | Hello | today | GZIP(GzipLevel(6)) | [PLAIN, RLE, BIT_PACKED] | | | 4 | 152 | 163 |
- +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+
+ +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------+-------------------+-----------------------+
+ | filename | row_group_id | row_group_num_rows | row_group_num_columns | row_group_bytes | column_id | file_offset | num_values | path_in_schema | type | stats_min | stats_max | stats_null_count | stats_distinct_count | stats_min_value | stats_max_value | compression | encodings | index_page_offset | dictionary_page_offset | data_page_offset | total_compressed_size | total_uncompressed_size | bloom_filter_offset | bloom_filter_length | column_index_offset | column_index_length | offset_index_offset | offset_index_length | sorting_columns | row_group_ordinal | row_group_file_offset |
+ +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------+-------------------+-----------------------+
+ | ../parquet-testing/data/data_index_bloom_encoding_stats.parquet | 0 | 14 | 1 | 163 | 0 | 4 | 14 | "String" | BYTE_ARRAY | Hello | today | 0 | | Hello | today | GZIP(GzipLevel(6)) | [PLAIN, RLE, BIT_PACKED] | | | 4 | 152 | 163 | 192 | | 156 | 25 | 181 | 11 | | 0 | 4 |
Review Comment:
The updated snapshots exercise several of the new footer fields (`bloom_filter_offset=192`, column/offset index offset+length, `row_group_ordinal`, `row_group_file_offset`), but `bloom_filter_length` and `sorting_columns` stay NULL/empty in both fixtures. That is expected for `data_index_bloom_encoding_stats.parquet` — `parquet-testing` documents it as having Bloom/page-index metadata without `bloom_filter_length`.
Because both `Some` paths stay unobserved, a regression that always pushed `None` for `bloom_filter_length` (lines 449–450) or `sorting_columns` (lines 457–458) would still pass.
`parquet-testing` already has fixtures for the missing cases:
- `data/data_index_bloom_encoding_with_length.parquet` — same file with `bloom_filter_length` populated
- `data/sort_columns.parquet` — two row groups with writer-declared sorting columns
Would it be useful to add a focused `SELECT`/snapshot against those files so extraction of the length (`i32` → `i64`) and `format!("{cols:?}")` formatting are actually locked in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: expose bloom filter, page index and sorting metadata in parquet… [datafusion]
Posted by "diegoQuinas (via GitHub)" <gi...@apache.org>.
diegoQuinas commented on code in PR #25638:
URL: https://github.com/apache/datafusion/pull/25638#discussion_r4097667660
##########
datafusion-cli/src/main.rs:
##########
@@ -599,11 +599,11 @@ mod tests {
let rbs = df.collect().await?;
assert_snapshot!(batches_to_string(&rbs),@r#"
- +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+
- | filename | row_group_id | row_group_num_rows | row_group_num_columns | row_group_bytes | column_id | file_offset | num_values | path_in_schema | type | stats_min | stats_max | stats_null_count | stats_distinct_count | stats_min_value | stats_max_value | compression | encodings | index_page_offset | dictionary_page_offset | data_page_offset | total_compressed_size | total_uncompressed_size |
- +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+
- | ../parquet-testing/data/data_index_bloom_encoding_stats.parquet | 0 | 14 | 1 | 163 | 0 | 4 | 14 | "String" | BYTE_ARRAY | Hello | today | 0 | | Hello | today | GZIP(GzipLevel(6)) | [PLAIN, RLE, BIT_PACKED] | | | 4 | 152 | 163 |
- +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+
+ +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------+-------------------+-----------------------+
+ | filename | row_group_id | row_group_num_rows | row_group_num_columns | row_group_bytes | column_id | file_offset | num_values | path_in_schema | type | stats_min | stats_max | stats_null_count | stats_distinct_count | stats_min_value | stats_max_value | compression | encodings | index_page_offset | dictionary_page_offset | data_page_offset | total_compressed_size | total_uncompressed_size | bloom_filter_offset | bloom_filter_length | column_index_offset | column_index_length | offset_index_offset | offset_index_length | sorting_columns | row_group_ordinal | row_group_file_offset |
+ +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------+-------------------+-----------------------+
+ | ../parquet-testing/data/data_index_bloom_encoding_stats.parquet | 0 | 14 | 1 | 163 | 0 | 4 | 14 | "String" | BYTE_ARRAY | Hello | today | 0 | | Hello | today | GZIP(GzipLevel(6)) | [PLAIN, RLE, BIT_PACKED] | | | 4 | 152 | 163 | 192 | | 156 | 25 | 181 | 11 | | 0 | 4 |
Review Comment:
>> Would it be useful to add a focused SELECT/snapshot against those files so extraction of the length (i32 → i64) and format!("{cols:?}") formatting are actually locked in?
Okay, of course. I'll add the focused snapshot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: expose bloom filter, page index and sorting metadata in parquet… [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25638:
URL: https://github.com/apache/datafusion/pull/25638#issuecomment-5787207951
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25638?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 82.45%. Comparing base ([`d5216e6`](https://app.codecov.io/gh/apache/datafusion/commit/d5216e60b85ee8afeca0f15ff148eee1e33eab35?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e97e875`](https://app.codecov.io/gh/apache/datafusion/commit/e97e87537f12565cd490581bd4318b832b90afd6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25638 +/- ##
==========================================
- Coverage 82.45% 82.45% -0.01%
==========================================
Files 1140 1140
Lines 436377 436377
Branches 436377 436377
==========================================
- Hits 359801 359797 -4
- Misses 54821 54823 +2
- Partials 21755 21757 +2
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25638?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: expose bloom filter, page index and sorting metadata in parquet… [datafusion]
Posted by "nuno-faria (via GitHub)" <gi...@apache.org>.
nuno-faria commented on code in PR #25638:
URL: https://github.com/apache/datafusion/pull/25638#discussion_r4238807909
##########
datafusion-cli/src/functions.rs:
##########
@@ -363,6 +363,15 @@ impl TableFunctionImpl for ParquetMetadataFunc {
Field::new("data_page_offset", DataType::Int64, true),
Field::new("total_compressed_size", DataType::Int64, true),
Field::new("total_uncompressed_size", DataType::Int64, true),
+ Field::new("bloom_filter_offset", DataType::Int64, true),
+ Field::new("bloom_filter_length", DataType::Int64, true),
+ Field::new("column_index_offset", DataType::Int64, true),
+ Field::new("column_index_length", DataType::Int64, true),
+ Field::new("offset_index_offset", DataType::Int64, true),
+ Field::new("offset_index_length", DataType::Int64, true),
+ Field::new("sorting_columns", DataType::Utf8, true),
+ Field::new("row_group_ordinal", DataType::Int64, true),
+ Field::new("row_group_file_offset", DataType::Int64, true),
Review Comment:
Since these metrics are based on the row group, it might be better to move them near to the other row group metrics (i.e., after `row_group_bytes`).
##########
datafusion-cli/src/functions.rs:
##########
@@ -430,6 +449,19 @@ impl TableFunctionImpl for ParquetMetadataFunc {
data_page_offset_arr.push(column.data_page_offset());
total_compressed_size_arr.push(column.compressed_size());
total_uncompressed_size_arr.push(column.uncompressed_size());
+ bloom_filter_offset_arr.push(column.bloom_filter_offset());
+ bloom_filter_length_arr
+ .push(column.bloom_filter_length().map(|v| v as i64));
+ column_index_offset_arr.push(column.column_index_offset());
+ column_index_length_arr
+ .push(column.column_index_length().map(|v| v as i64));
+ offset_index_offset_arr.push(column.offset_index_offset());
+ offset_index_length_arr
+ .push(column.offset_index_length().map(|v| v as i64));
+ sorting_columns_arr
+ .push(row_group.sorting_columns().map(|cols| format!("{cols:?}")));
+ row_group_ordinal_arr.push(row_group.ordinal().map(|v| v as i64));
Review Comment:
Is there a difference between the ordinal and the existing row_group_id?
##########
datafusion-cli/src/main.rs:
##########
@@ -609,11 +609,51 @@ mod tests {
let rbs = df.collect().await?;
assert_snapshot!(batches_to_string(&rbs),@r#"
- +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+
- | filename | row_group_id | row_group_num_rows | row_group_num_columns | row_group_bytes | column_id | file_offset | num_values | path_in_schema | type | stats_min | stats_max | stats_null_count | stats_distinct_count | stats_min_value | stats_max_value | compression | encodings | index_page_offset | dictionary_page_offset | data_page_offset | total_compressed_size | total_uncompressed_size |
- +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+
- | ../parquet-testing/data/data_index_bloom_encoding_stats.parquet | 0 | 14 | 1 | 163 | 0 | 4 | 14 | "String" | BYTE_ARRAY | Hello | today | 0 | | Hello | today | GZIP(GzipLevel(6)) | [PLAIN, RLE, BIT_PACKED] | | | 4 | 152 | 163 |
- +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+
+ +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------+-------------------+-----------------------+
+ | filename | row_group_id | row_group_num_rows | row_group_num_columns | row_group_bytes | column_id | file_offset | num_values | path_in_schema | type | stats_min | stats_max | stats_null_count | stats_distinct_count | stats_min_value | stats_max_value | compression | encodings | index_page_offset | dictionary_page_offset | data_page_offset | total_compressed_size | total_uncompressed_size | bloom_filter_offset | bloom_filter_length | column_index_offset | column_index_length | offset_index_offset | offset_index_length | sorting_columns | row_group_ordinal | row_group_file_offset |
+ +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------+-------------------+-----------------------+
+ | ../parquet-testing/data/data_index_bloom_encoding_stats.parquet | 0 | 14 | 1 | 163 | 0 | 4 | 14 | "String" | BYTE_ARRAY | Hello | today | 0 | | Hello | today | GZIP(GzipLevel(6)) | [PLAIN, RLE, BIT_PACKED] | | | 4 | 152 | 163 | 192 | | 156 | 25 | 181 | 11 | | 0 | 4 |
+ +-----------------------------------------------------------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+--------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+-----------------+-------------------+-----------------------+
+ "#);
+
+ Ok(())
+ }
+
+ /// `bloom_filter_length` and `sorting_columns` are absent from the other
+ /// fixtures. These files populate them so a regression that always emits
+ /// NULL still fails.
+ #[tokio::test]
+ async fn test_parquet_metadata_bloom_length_and_sorting_columns()
+ -> Result<(), DataFusionError> {
+ let ctx = SessionContext::new();
+ ctx.register_udtf("parquet_metadata", Arc::new(ParquetMetadataFunc {}));
+
+ let sql = "SELECT path_in_schema, bloom_filter_offset, bloom_filter_length \
+ FROM parquet_metadata('../parquet-testing/data/data_index_bloom_encoding_with_length.parquet')";
+ let df = ctx.sql(sql).await?;
+ let rbs = df.collect().await?;
+ assert_snapshot!(batches_to_string(&rbs), @r#"
+ +----------------+---------------------+---------------------+
+ | path_in_schema | bloom_filter_offset | bloom_filter_length |
+ +----------------+---------------------+---------------------+
+ | "String" | 253 | 2064 |
+ +----------------+---------------------+---------------------+
+ "#);
+
+ let sql = "SELECT row_group_id, column_id, path_in_schema, sorting_columns \
+ FROM parquet_metadata('../parquet-testing/data/sort_columns.parquet') \
+ ORDER BY row_group_id, column_id";
+ let df = ctx.sql(sql).await?;
+ let rbs = df.collect().await?;
+ assert_snapshot!(batches_to_string(&rbs), @r#"
+ +--------------+-----------+----------------+------------------------------------------------------------------------------------------------------------------------------------------------+
+ | row_group_id | column_id | path_in_schema | sorting_columns |
+ +--------------+-----------+----------------+------------------------------------------------------------------------------------------------------------------------------------------------+
+ | 0 | 0 | "a" | [SortingColumn { column_idx: 0, descending: true, nulls_first: true }, SortingColumn { column_idx: 1, descending: false, nulls_first: false }] |
Review Comment:
I think the current `sorting_columns` is difficult to read. To me at least I would prefer the column names instead. E.g.:
`[SortingColumn { column_idx: 0, descending: true, nulls_first: true }, SortingColumn { column_idx: 1, descending: false, nulls_first: false }]`
vs
`col1 DESC NULLS FIRST, col2 ASC`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org