You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "etseidl (via GitHub)" <gi...@apache.org> on 2025/08/29 18:54:39 UTC

[PR] [Minor] Backport changes to metadata benchmark [arrow-rs]

etseidl opened a new pull request, #8251:
URL: https://github.com/apache/arrow-rs/pull/8251

   # Which issue does this PR close?
   
   - Part of #5854.
   
   # Rationale for this change
   Backport changes to allow apples-to-apples comparison of thrift decoding
   
   # What changes are included in this PR?
   
   Adds a page header benchmark and updates bench names to match those in feature branch.
   
   # Are these changes tested?
   
   No tests needed...only changes to benchmark
   
   # Are there any user-facing changes?
   
   No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Minor] Backport changes to metadata benchmark [arrow-rs]

Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #8251:
URL: https://github.com/apache/arrow-rs/pull/8251#discussion_r2324994127


##########
parquet/src/thrift.rs:
##########
@@ -33,12 +33,18 @@ pub trait TSerializable: Sized {
     fn write_to_out_protocol<T: TOutputProtocol>(&self, o_prot: &mut T) -> thrift::Result<()>;
 }
 
-/// Public function to aid benchmarking.
+/// Public function to aid benchmarking. Reads Parquet `FileMetaData` encoded in `bytes`.
 pub fn bench_file_metadata(bytes: &bytes::Bytes) {
     let mut input = TCompactSliceInputProtocol::new(bytes);
     crate::format::FileMetaData::read_from_in_protocol(&mut input).unwrap();
 }
 
+/// Public function to aid benchmarking. Reads Parquet `PageHeader` encoded in `bytes`.
+pub fn bench_page_header(bytes: &bytes::Bytes) {

Review Comment:
   Maybe we can put these functions behind a feature flag and/or add `#[doc(hidden)]`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Minor] Backport changes to metadata benchmark [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8251:
URL: https://github.com/apache/arrow-rs/pull/8251#discussion_r2312149350


##########
parquet/benches/metadata.rs:
##########
@@ -93,7 +94,18 @@ fn encoded_meta() -> Vec<u8> {
                         index_page_offset: Some(rng.random()),
                         dictionary_page_offset: Some(rng.random()),
                         statistics: Some(stats.clone()),
-                        encoding_stats: None,
+                        encoding_stats: Some(vec![

Review Comment:
   Adding some encoding stats because profiling showed around 25% of the time to read the file metadata struct is spent reading this vector 😮 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Minor] Backport changes to metadata benchmark [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8251:
URL: https://github.com/apache/arrow-rs/pull/8251#issuecomment-3237932681

   With these changes, I can again use critcmp
   
   ```
   % critcmp main read_page_header
   group                                 main                                   read_page_header
   -----                                 ----                                   ----------------
   decode parquet metadata               1.87     35.8±0.98µs        ? ?/sec    1.00     19.1±0.33µs        ? ?/sec
   decode parquet metadata (wide)        2.73    221.4±2.90ms        ? ?/sec    1.00     81.0±2.87ms        ? ?/sec
   decode thrift file metadata           2.22     21.8±0.31µs        ? ?/sec    1.00      9.8±0.20µs        ? ?/sec
   decode thrift file metadata (wide)    1.80    112.7±2.39ms        ? ?/sec    1.00     62.8±1.27ms        ? ?/sec
   open(default)                         1.79     36.2±1.00µs        ? ?/sec    1.00     20.2±0.49µs        ? ?/sec
   open(page index)                      6.80  1773.7±23.97µs        ? ?/sec    1.00    260.7±4.39µs        ? ?/sec
   page headers                          1.67     12.1±0.27µs        ? ?/sec    1.00      7.3±0.17µs        ? ?/sec
   page headers (no stats)                                                      1.00      4.2±0.06µs        ? ?/sec
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Minor] Backport changes to metadata benchmark [arrow-rs]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8251:
URL: https://github.com/apache/arrow-rs/pull/8251#issuecomment-3259275115

   Thanks @mbrobbel!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Minor] Backport changes to metadata benchmark [arrow-rs]

Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel merged PR #8251:
URL: https://github.com/apache/arrow-rs/pull/8251


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org