You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "mkleen (via GitHub)" <gi...@apache.org> on 2026/09/29 09:23:51 UTC
[PR] bench(parquet): add page index statistics decoding benchmark [arrow-rs]
mkleen opened a new pull request, #11287:
URL: https://github.com/apache/arrow-rs/pull/11287
# Which issue does this PR close?
- Part of #9296.
- Pre-requisite for https://github.com/apache/arrow-rs/pull/11285
# Rationale for this change
Decoding Parquet statistics and converting them to Arrow can take as long as running the query itself.
For page pruning, DataFusion's `page_filter.rs` asks `StatisticsConverter` for the data page mins, maxes and null counts of every row group it scans. Each request goes through the fully parsed `ColumnIndexMetaData`, so the column index is decoded into `Vec`s first and then copied into Arrow arrays.
I'm planning a [follow-up PR]( https://github.com/apache/arrow-rs/pull/11285) that decodes the stored `ColumnIndex` bytes straight into Arrow arrays, skipping the intermediate `ColumnIndexMetaData`.
This is a the baseline benchmark for this pr.
# What changes are included in this PR?
A new `page_index_benchmark` group in `parquet/benches/arrow_statistics.rs`:
- Writes an in-memory file with 20 row groups, 10 rows per data page and page-level statistics, with every 7th value null.
- Covers `Int64`, `Utf8`, `Utf8View` and `Decimal128(20, 2)`, each with 2,000 and 10,000 data pages in total.
- Reads each row group's raw column index bytes (via `column_index_range()`) and, inside the timed loop, does what DataFusion's page pruning effectively does today: `decode_column_index` → `PageIndexBuilder` → `StatisticsConverter::data_page_{mins,maxes,null_counts,nan_counts}`.
# Are these changes tested?
No.
# Are there any user-facing changes?
No.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bench(parquet): Add page index statistics decoding benchmark [arrow-rs]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #11287:
URL: https://github.com/apache/arrow-rs/pull/11287#issuecomment-5905061916
@etseidl This is ready to merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bench(parquet): Add page index statistics decoding benchmark [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #11287:
URL: https://github.com/apache/arrow-rs/pull/11287#issuecomment-5904506503
@mkleen let me know when you think this is ready to merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bench(parquet): Add page index statistics decoding benchmark [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl merged PR #11287:
URL: https://github.com/apache/arrow-rs/pull/11287
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bench(parquet): Add page index statistics decoding benchmark [arrow-rs]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #11287:
URL: https://github.com/apache/arrow-rs/pull/11287#issuecomment-5905155187
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org