You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "haohuaijin (via GitHub)" <gi...@apache.org> on 2026/09/06 13:26:38 UTC
[PR] bench(parquet): add same-projection predicate chain benchmark [arrow-rs]
haohuaijin opened a new pull request, #11007:
URL: https://github.com/apache/arrow-rs/pull/11007
# Which issue does this PR close?
Related to #10926.
# Rationale for this change
`RowFilter` evaluates each `ArrowPredicate` separately, so consecutive predicates on the same projection decode that column, or replay it from the predicate cache, once per predicate. #10859 fuses such chains. Per the contributing guide, the benchmark is submitted separately so it can run on the automated runner and serve as the baseline for that change.
# What changes are included in this PR?
A criterion benchmark, `parquet/benches/arrow_reader_predicate_fusion.rs`, that scans an in-memory Snappy Parquet file of 262,144 rows through the async reader. Case names are `type/layout/cache/predicates/profile`:
- `int64` / `string` filter column, `fragmented` (one key per row) or `clustered` (each key repeated for 128 rows)
- `cached` / `uncached`: whether the filter column is also projected and therefore served by the predicate cache
- chains of 1 (control), 2 and 4 predicates
- survivor profiles `all99`, `all50`, `early1` and `late1`
- a `selection_boundary` group with run lengths 16, 32 and 64 around the default row selection policy threshold
75 cases in total. Each case is validated once outside measurement for the expected row count and predicate cache use.
# Are these changes tested?
The benchmark builds and its registration-time validation passes on `main`.
# Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bench(parquet): add same-projection predicate chain benchmark [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11007:
URL: https://github.com/apache/arrow-rs/pull/11007#issuecomment-5735999803
This looks good except it makes 75 benchmarks
```shell
cargo bench --bench arrow_reader_predicate_fusion --features="arrow async snap"
...
.00%) high severe
same_projection_filter/string/clustered/cached/1/all99
time: [1.6653 ms 1.6686 ms 1.6722 ms]
thrpt: [156.77 Melem/s 157.11 Melem/s 157.42 Melem/s]
Found 10 outliers among 100 measurements (10.00%)
4 (4.00%) high mild
6 (6.00%) high severe
same_projection_filter/string/clustered/cached/2/all99
time: [2.5275 ms 2.5337 ms 2.5405 ms]
thrpt: [103.18 Melem/s 103.46 Melem/s 103.72 Melem/s]
Found 4 outliers among 100 measurements (4.00%)
3 (3.00%) high mild
1 (1.00%) high severe
same_projection_filter/string/clustered/cached/2/all50
time: [2.8156 ms 2.8271 ms 2.8381 ms]
thrpt: [92.366 Melem/s 92.725 Melem/s 93.104 Melem/s]
Found 17 outliers among 100 measurements (17.00%)
11 (11.00%) low severe
6 (6.00%) high severe
same_projection_filter/string/clustered/cached/2/early1
time: [1.3208 ms 1.3229 ms 1.3253 ms]
thrpt: [197.80 Melem/s 198.16 Melem/s 198.47 Melem/s]
Found 8 outliers among 100 measurements (8.00%)
6 (6.00%) high mild
2 (2.00%) high severe
same_projection_filter/string/clustered/cached/2/late1
time: [2.0176 ms 2.0224 ms 2.0277 ms]
thrpt: [129.28 Melem/s 129.62 Melem/s 129.93 Melem/s]
Found 17 outliers among 100 measurements (17.00%)
1 (1.00%) high mild
16 (16.00%) high severe
same_projection_filter/string/clustered/cached/4/all99
time: [4.3241 ms 4.3359 ms 4.3489 ms]
thrpt: [60.278 Melem/s 60.459 Melem/s 60.624 Melem/s]
Found 8 outliers among 100 measurements (8.00%)
5 (5.00%) high mild
3 (3.00%) high severe
same_projection_filter/string/clustered/cached/4/all50
time: [3.3406 ms 3.3495 ms 3.3579 ms]
thrpt: [78.068 Melem/s 78.263 Melem/s 78.472 Melem/s]
Found 11 outliers among 100 measurements (11.00%)
3 (3.00%) low severe
8 (8.00%) high severe
same_projection_filter/string/clustered/cached/4/early1
time: [1.4078 ms 1.4143 ms 1.4211 ms]
thrpt: [184.46 Melem/s 185.35 Melem/s 186.21 Melem/s]
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
same_projection_filter/string/clustered/cached/4/late1
time: [3.8255 ms 3.8473 ms 3.8698 ms]
thrpt: [67.742 Melem/s 68.138 Melem/s 68.526 Melem/s]
Found 4 outliers among 100 measurements (4.00%)
4 (4.00%) high mild
same_projection_filter/selection_boundary/int64/uncached/4/run16
time: [6.3433 ms 6.3583 ms 6.3741 ms]
thrpt: [41.127 Melem/s 41.229 Melem/s 41.326 Melem/s]
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
same_projection_filter/selection_boundary/int64/uncached/4/run32
time: [6.0143 ms 6.0268 ms 6.0399 ms]
thrpt: [43.402 Melem/s 43.496 Melem/s 43.587 Melem/s]
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
same_projection_filter/selection_boundary/int64/uncached/4/run64
time: [5.7467 ms 5.7677 ms 5.7904 ms]
thrpt: [45.272 Melem/s 45.451 Melem/s 45.617 Melem/s]
Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) high mild
1 (1.00%) high severe
```
Do we really need them all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bench(parquet): add same-projection predicate chain benchmark [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11007:
URL: https://github.com/apache/arrow-rs/pull/11007#issuecomment-5740878580
Thanks @haohuaijin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bench(parquet): add same-projection predicate chain benchmark [arrow-rs]
Posted by "haohuaijin (via GitHub)" <gi...@apache.org>.
haohuaijin commented on PR #11007:
URL: https://github.com/apache/arrow-rs/pull/11007#issuecomment-5739261106
> Do we really need them all?
Good point! i added 75 cases because i want to have good cover for benchmark, but it can be reduce. I've reduce the case down from 75 to 30 cases while keeping the main scenarios covered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bench(parquet): add same-projection predicate chain benchmark [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #11007:
URL: https://github.com/apache/arrow-rs/pull/11007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org