You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "etseidl (via GitHub)" <gi...@apache.org> on 2025/12/16 19:52:52 UTC
[PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
etseidl opened a new pull request, #9008:
URL: https://github.com/apache/arrow-rs/pull/9008
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.
-->
- Part of #5853.
# Rationale for this change
Add ability to skip the decoding of more types of statistics contained in the Parquet column metadata. While this currently doesn't have a huge impact on decode time, it can reduce the amount of memory used by the `ParquetMetaData`.
# What changes are included in this PR?
Adds more options and tests for those options. Also adds size statistics to the metadata bench.
# Are these changes tested?
Yes
# Are there any user-facing changes?
Only adds new options, no breaking changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3730538628
This PR appears to have picked up some conflicts -- likely with
- https://github.com/apache/arrow-rs/pull/9051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3719737504
Thanks @etseidl -- let's wait then -- we already have quite a lot of good stuff queued up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3676486715
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main skip_column_stats
----- ---- -----------------
decoding: dtype=FixedLenByteArray(16), encoding=BYTE_STREAM_SPLIT 1.01 560.7±6.71µs ? ?/sec 1.00 557.3±4.57µs ? ?/sec
decoding: dtype=FixedLenByteArray(2), encoding=BYTE_STREAM_SPLIT 1.00 371.3±4.78µs ? ?/sec 1.03 383.7±15.37µs ? ?/sec
decoding: dtype=f32, encoding=BYTE_STREAM_SPLIT 1.00 34.4±0.21µs ? ?/sec 1.00 34.5±0.45µs ? ?/sec
decoding: dtype=f64, encoding=BYTE_STREAM_SPLIT 1.00 105.0±1.28µs ? ?/sec 1.00 105.2±1.16µs ? ?/sec
encoding: dtype=FixedLenByteArray(16), encoding=BYTE_STREAM_SPLIT 1.00 325.4±3.17µs ? ?/sec 1.00 325.0±3.15µs ? ?/sec
encoding: dtype=FixedLenByteArray(2), encoding=BYTE_STREAM_SPLIT 1.05 44.2±6.92µs ? ?/sec 1.00 42.1±5.30µs ? ?/sec
encoding: dtype=f32, encoding=BYTE_STREAM_SPLIT 1.00 22.9±0.16µs ? ?/sec 1.00 22.9±0.06µs ? ?/sec
encoding: dtype=f64, encoding=BYTE_STREAM_SPLIT 1.00 56.9±0.27µs ? ?/sec 1.00 57.0±0.34µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3730539989
main is now open for the next release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3676509441
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group main skip_column_stats
----- ---- -----------------
decode metadata (wide) with schema 1.02 40.1±0.31ms ? ?/sec 1.00 39.5±0.42ms ? ?/sec
decode metadata (wide) with skip PES 1.04 40.1±0.70ms ? ?/sec 1.00 38.7±0.25ms ? ?/sec
decode metadata (wide) with skip all stats 1.00 44.2±0.44ms ? ?/sec
decode metadata (wide) with skip column stats 1.00 42.6±0.50ms ? ?/sec
decode metadata (wide) with skip size stats 1.00 49.0±1.01ms ? ?/sec
decode metadata (wide) with stats mask 1.02 39.8±0.33ms ? ?/sec 1.00 38.9±0.40ms ? ?/sec
decode metadata with schema 1.01 5.7±0.08µs ? ?/sec 1.00 5.6±0.03µs ? ?/sec
decode metadata with skip PES 1.02 8.9±0.06µs ? ?/sec 1.00 8.7±0.07µs ? ?/sec
decode metadata with skip column stats 1.00 8.9±0.13µs ? ?/sec
decode metadata with stats mask 1.02 8.9±0.14µs ? ?/sec 1.00 8.8±0.10µs ? ?/sec
decode parquet metadata 1.00 9.4±0.11µs ? ?/sec 1.00 9.4±0.08µs ? ?/sec
decode parquet metadata (wide) 1.02 43.4±0.39ms ? ?/sec 1.00 42.4±0.43ms ? ?/sec
decode parquet metadata w/ size stats (wide) 1.00 55.8±0.65ms ? ?/sec
open(default) 1.02 10.0±0.20µs ? ?/sec 1.00 9.8±0.11µs ? ?/sec
open(page index) 1.00 166.1±1.37µs ? ?/sec 1.00 165.9±0.71µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3676467422
run benchmark encoding metadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3732547705
Thanks again @etseidl and @scovich
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3676486817
🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing skip_column_stats (072ecf62ef7e78cb22423fc97149f0e5301672fc) to c2bd7d91bfb9e835daff07e41f194c0d3e53d858 [diff](https://github.com/apache/arrow-rs/compare/c2bd7d91bfb9e835daff07e41f194c0d3e53d858..072ecf62ef7e78cb22423fc97149f0e5301672fc)
BENCH_NAME=metadata
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench metadata
BENCH_FILTER=
BENCH_BRANCH_NAME=skip_column_stats
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3676333895
Sorry I didn't see this one before. I'll try and review it shortly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3732547563
> @alamb I'm finally waking from my holiday hibernation.
Nice! That was me too this week (and there was quite a bit of backlog to get through!)
I spent some non trivial amount of time over the break studying and profiling the parquet reader so I expect to be doing a bunch of micro optimizations there (aka reducing allocations / reallocations).
Should be a fun January!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3716693600
> I am working to prepare the arrow 57.2.0 release -- do we want to merge this one, or shall we wait for arrow 58?
I'll defer to your judgement @alamb. I don't think there's any pressing need at the moment, but it also doesn't need any further changes AFAIK, and it's not a breaking change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3676467673
🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing skip_column_stats (072ecf62ef7e78cb22423fc97149f0e5301672fc) to c2bd7d91bfb9e835daff07e41f194c0d3e53d858 [diff](https://github.com/apache/arrow-rs/compare/c2bd7d91bfb9e835daff07e41f194c0d3e53d858..072ecf62ef7e78cb22423fc97149f0e5301672fc)
BENCH_NAME=encoding
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench encoding
BENCH_FILTER=
BENCH_BRANCH_NAME=skip_column_stats
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3730834466
@alamb I'm finally waking from my holiday hibernation. I fixed the conflicts, but also reverted some benchmark changes made in #9051 that I didn't have the opportunity to review. The non-new benches should now match their counterparts in `57.2`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3716488704
I am working to prepare the arrow 57.2.0 release -- do we want to merge this one, or shall we wait for arrow 58?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org