You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "etseidl (via GitHub)" <gi...@apache.org> on 2025/12/16 19:52:52 UTC

[PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

etseidl opened a new pull request, #9008:
URL: https://github.com/apache/arrow-rs/pull/9008

   # Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.
   -->
   
   - Part of #5853.
   
   # Rationale for this change
   
   
   
   Add ability to skip the decoding of more types of statistics contained in the Parquet column metadata. While this currently doesn't have a huge impact on decode time, it can reduce the amount of memory used by the `ParquetMetaData`.
   
   # What changes are included in this PR?
   
   Adds more options and tests for those options. Also adds size statistics to the metadata bench.
   
   # Are these changes tested?
   
   Yes
   
   # Are there any user-facing changes?
   Only adds new options, no breaking changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3730538628

   This PR appears to have picked up some conflicts -- likely with 
   - https://github.com/apache/arrow-rs/pull/9051


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3719737504

   Thanks @etseidl  -- let's wait then -- we already have quite a lot of good stuff queued up


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3676486715

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                                                main                                   skip_column_stats
   -----                                                                ----                                   -----------------
   decoding: dtype=FixedLenByteArray(16), encoding=BYTE_STREAM_SPLIT    1.01    560.7±6.71µs        ? ?/sec    1.00    557.3±4.57µs        ? ?/sec
   decoding: dtype=FixedLenByteArray(2), encoding=BYTE_STREAM_SPLIT     1.00    371.3±4.78µs        ? ?/sec    1.03   383.7±15.37µs        ? ?/sec
   decoding: dtype=f32, encoding=BYTE_STREAM_SPLIT                      1.00     34.4±0.21µs        ? ?/sec    1.00     34.5±0.45µs        ? ?/sec
   decoding: dtype=f64, encoding=BYTE_STREAM_SPLIT                      1.00    105.0±1.28µs        ? ?/sec    1.00    105.2±1.16µs        ? ?/sec
   encoding: dtype=FixedLenByteArray(16), encoding=BYTE_STREAM_SPLIT    1.00    325.4±3.17µs        ? ?/sec    1.00    325.0±3.15µs        ? ?/sec
   encoding: dtype=FixedLenByteArray(2), encoding=BYTE_STREAM_SPLIT     1.05     44.2±6.92µs        ? ?/sec    1.00     42.1±5.30µs        ? ?/sec
   encoding: dtype=f32, encoding=BYTE_STREAM_SPLIT                      1.00     22.9±0.16µs        ? ?/sec    1.00     22.9±0.06µs        ? ?/sec
   encoding: dtype=f64, encoding=BYTE_STREAM_SPLIT                      1.00     56.9±0.27µs        ? ?/sec    1.00     57.0±0.34µs        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3730539989

   main is now open for the next release


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3676509441

   🤖: Benchmark completed
   
   <details><summary>Details</summary>
   <p>
   
   
   ```
   group                                            main                                   skip_column_stats
   -----                                            ----                                   -----------------
   decode metadata (wide) with schema               1.02     40.1±0.31ms        ? ?/sec    1.00     39.5±0.42ms        ? ?/sec
   decode metadata (wide) with skip PES             1.04     40.1±0.70ms        ? ?/sec    1.00     38.7±0.25ms        ? ?/sec
   decode metadata (wide) with skip all stats                                              1.00     44.2±0.44ms        ? ?/sec
   decode metadata (wide) with skip column stats                                           1.00     42.6±0.50ms        ? ?/sec
   decode metadata (wide) with skip size stats                                             1.00     49.0±1.01ms        ? ?/sec
   decode metadata (wide) with stats mask           1.02     39.8±0.33ms        ? ?/sec    1.00     38.9±0.40ms        ? ?/sec
   decode metadata with schema                      1.01      5.7±0.08µs        ? ?/sec    1.00      5.6±0.03µs        ? ?/sec
   decode metadata with skip PES                    1.02      8.9±0.06µs        ? ?/sec    1.00      8.7±0.07µs        ? ?/sec
   decode metadata with skip column stats                                                  1.00      8.9±0.13µs        ? ?/sec
   decode metadata with stats mask                  1.02      8.9±0.14µs        ? ?/sec    1.00      8.8±0.10µs        ? ?/sec
   decode parquet metadata                          1.00      9.4±0.11µs        ? ?/sec    1.00      9.4±0.08µs        ? ?/sec
   decode parquet metadata (wide)                   1.02     43.4±0.39ms        ? ?/sec    1.00     42.4±0.43ms        ? ?/sec
   decode parquet metadata w/ size stats (wide)                                            1.00     55.8±0.65ms        ? ?/sec
   open(default)                                    1.02     10.0±0.20µs        ? ?/sec    1.00      9.8±0.11µs        ? ?/sec
   open(page index)                                 1.00    166.1±1.37µs        ? ?/sec    1.00    165.9±0.71µs        ? ?/sec
   ```
   
   
   </p>
   </details>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3676467422

   run benchmark encoding metadata


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3732547705

   Thanks again @etseidl and @scovich 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3676486817

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing skip_column_stats (072ecf62ef7e78cb22423fc97149f0e5301672fc) to c2bd7d91bfb9e835daff07e41f194c0d3e53d858 [diff](https://github.com/apache/arrow-rs/compare/c2bd7d91bfb9e835daff07e41f194c0d3e53d858..072ecf62ef7e78cb22423fc97149f0e5301672fc)
   BENCH_NAME=metadata
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench metadata 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=skip_column_stats
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3676333895

   Sorry I didn't see this one before. I'll try and review it shortly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3732547563

   > @alamb I'm finally waking from my holiday hibernation. 
   
   Nice! That was me too this week (and there was quite a bit of backlog to get through!)
   
   I spent some non trivial amount of time over the break studying and profiling the parquet reader so I expect to be doing a bunch of micro optimizations there (aka reducing allocations / reallocations). 
   
   Should be a fun January!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3716693600

   > I am working to prepare the arrow 57.2.0 release -- do we want to merge this one, or shall we wait for arrow 58?
   
   I'll defer to your judgement @alamb. I don't think there's any pressing need at the moment, but it also doesn't need any further changes AFAIK, and it's not a breaking change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "alamb-ghbot (via GitHub)" <gi...@apache.org>.
alamb-ghbot commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3676467673

   🤖 `./gh_compare_arrow.sh` [gh_compare_arrow.sh](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
   Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   Comparing skip_column_stats (072ecf62ef7e78cb22423fc97149f0e5301672fc) to c2bd7d91bfb9e835daff07e41f194c0d3e53d858 [diff](https://github.com/apache/arrow-rs/compare/c2bd7d91bfb9e835daff07e41f194c0d3e53d858..072ecf62ef7e78cb22423fc97149f0e5301672fc)
   BENCH_NAME=encoding
   BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench encoding 
   BENCH_FILTER=
   BENCH_BRANCH_NAME=skip_column_stats
   Results will be posted here when complete
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3730834466

   @alamb I'm finally waking from my holiday hibernation. I fixed the conflicts, but also reverted some benchmark changes made in #9051 that I didn't have the opportunity to review. The non-new benches should now match their counterparts in `57.2`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add options to skip decoding `Statistics` and `SizeStatistics` in Parquet metadata [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9008:
URL: https://github.com/apache/arrow-rs/pull/9008#issuecomment-3716488704

   I am working to prepare the arrow 57.2.0 release -- do we want to merge this one, or shall we wait for arrow 58?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org