You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/05/01 01:45:09 UTC
[PR] Update arrow_reader_row_filter benchmark to reflect ClickBench distribution [arrow-rs]
alamb opened a new pull request, #7461:
URL: https://github.com/apache/arrow-rs/pull/7461
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes https://github.com/apache/arrow-rs/issues/7460
# Rationale for this change
We would like a benchmark that accurately reflects the predicate patterns we see in ClickBench.
See full analysis here https://github.com/apache/arrow-rs/issues/7460#issuecomment-2843142141
ClickHouse Data in `hits.parquet`:
* Selectivity is: `13172392` / `99997497` = `0.132`
* Number of `RowSelections` = `14054784`
* Average run length of each `RowSelection`: `99997497` / `14054784` = `7.114`
Data in the arrow_reader_row_filter benchmark:
* Selectivity is: `80147` / `100000` = `0.8`
* Number of RowSelections = `67989`
* Average run length of each RowSelection: `100000 / 32010` = `3.1`
# What changes are included in this PR?
Change distribution of the string view so it is much closer to parquet data (the selectivity especially)
* Selectivity is: `15144` / `100000` = `0.15144`
* Number of RowSelections = `12904`
* Average run length of each RowSelection: `100000` / `12904` = 7.75
# Are there any user-facing changes?
No, this is a benchmark only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update arrow_reader_row_filter benchmark to reflect ClickBench distribution [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7461:
URL: https://github.com/apache/arrow-rs/pull/7461#issuecomment-2843904300
Unfortunately, even after adjusting the benchmark on this branch I still don't see major changes in https://github.com/apache/arrow-rs/pull/7428.
I will look more deeply tomorrow
```shell
cargo bench --all-features --bench arrow_reader_row_filter -- Utf8ViewNonEmpty
```
Main compared to https://github.com/apache/arrow-rs/pull/7428
```
arrow_reader_row_filter/Utf8ViewNonEmpty/all_columns/async
time: [4.1253 ms 4.1553 ms 4.1881 ms]
change: [-4.8097% -3.9190% -3.0939%] (p = 0.00 < 0.05)
Performance has improved.
Found 19 outliers among 100 measurements (19.00%)
14 (14.00%) high mild
5 (5.00%) high severe
arrow_reader_row_filter/Utf8ViewNonEmpty/all_columns/sync
time: [4.2269 ms 4.2340 ms 4.2419 ms]
change: [-1.5246% -1.1130% -0.7616%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
3 (3.00%) high mild
2 (2.00%) high severe
arrow_reader_row_filter/Utf8ViewNonEmpty/exclude_filter_column/async
time: [3.0754 ms 3.0802 ms 3.0857 ms]
change: [-3.2754% -2.7568% -2.2574%] (p = 0.00 < 0.05)
Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
6 (6.00%) high mild
5 (5.00%) high severe
arrow_reader_row_filter/Utf8ViewNonEmpty/exclude_filter_column/sync
time: [3.0774 ms 3.0839 ms 3.0909 ms]
change: [-1.4528% -1.1133% -0.7921%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
6 (6.00%) high mild
1 (1.00%) high severe
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update arrow_reader_row_filter benchmark to reflect ClickBench distribution [arrow-rs]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #7461:
URL: https://github.com/apache/arrow-rs/pull/7461#issuecomment-2847187119
> > Thank you @alamb for this work, so we still need to investigate more. Is it possible that we can get a 10% data set from hit.parquet and do some benchmark from arrow-rs side.
>
> That is an interesting idea -- to make make a benchmark in arrow-rs that runs against hits.parquet (and hits_partitioned) directly 🤔 (and e.g. could require downloading those files before running).
I am trying to do the first step, may be we can download a partition hit.parquet, and pick it as the data set to arrow-rs, because we have 100 partition file, it seems about %1 data which can be mocked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update arrow_reader_row_filter benchmark to reflect ClickBench distribution [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7461:
URL: https://github.com/apache/arrow-rs/pull/7461#issuecomment-2852157245
> I have been thinking about it too -- I probably won't have a chance this weekend but can work on it next week
- I started one here: https://github.com/apache/arrow-rs/pull/7470
Will keep at it tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update arrow_reader_row_filter benchmark to reflect ClickBench distribution [arrow-rs]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #7461:
URL: https://github.com/apache/arrow-rs/pull/7461#issuecomment-2844151896
> Unfortunately, even after adjusting the benchmark on this branch I still don't see major changes in #7428.
>
> I will look more deeply tomorrow
>
> ```shell
> cargo bench --all-features --bench arrow_reader_row_filter -- Utf8ViewNonEmpty
> ```
>
> Main compared to #7428
>
> ```
> arrow_reader_row_filter/Utf8ViewNonEmpty/all_columns/async
> time: [4.1253 ms 4.1553 ms 4.1881 ms]
> change: [-4.8097% -3.9190% -3.0939%] (p = 0.00 < 0.05)
> Performance has improved.
> Found 19 outliers among 100 measurements (19.00%)
> 14 (14.00%) high mild
> 5 (5.00%) high severe
> arrow_reader_row_filter/Utf8ViewNonEmpty/all_columns/sync
> time: [4.2269 ms 4.2340 ms 4.2419 ms]
> change: [-1.5246% -1.1130% -0.7616%] (p = 0.00 < 0.05)
> Change within noise threshold.
> Found 5 outliers among 100 measurements (5.00%)
> 3 (3.00%) high mild
> 2 (2.00%) high severe
> arrow_reader_row_filter/Utf8ViewNonEmpty/exclude_filter_column/async
> time: [3.0754 ms 3.0802 ms 3.0857 ms]
> change: [-3.2754% -2.7568% -2.2574%] (p = 0.00 < 0.05)
> Performance has improved.
> Found 11 outliers among 100 measurements (11.00%)
> 6 (6.00%) high mild
> 5 (5.00%) high severe
> arrow_reader_row_filter/Utf8ViewNonEmpty/exclude_filter_column/sync
> time: [3.0774 ms 3.0839 ms 3.0909 ms]
> change: [-1.4528% -1.1133% -0.7921%] (p = 0.00 < 0.05)
> Change within noise threshold.
> Found 7 outliers among 100 measurements (7.00%)
> 6 (6.00%) high mild
> 1 (1.00%) high severe
> ```
Thank you @alamb for this work, so we still need to investigate more. Is it possible that we can get a 10% data set from hit.parquet and do some benchmark from arrow-rs side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update arrow_reader_row_filter benchmark to reflect ClickBench distribution [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7461:
URL: https://github.com/apache/arrow-rs/pull/7461#issuecomment-2844751160
> Thank you @alamb for this work, so we still need to investigate more. Is it possible that we can get a 10% data set from hit.parquet and do some benchmark from arrow-rs side.
That is an interesting idea -- to make make a benchmark in arrow-rs that runs against hits.parquet (and hits_partitioned) directly 🤔 (and e.g. could require downloading those files before running).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update arrow_reader_row_filter benchmark to reflect ClickBench distribution [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7461:
URL: https://github.com/apache/arrow-rs/pull/7461#issuecomment-2848051135
> > > Thank you @alamb for this work, so we still need to investigate more. Is it possible that we can get a 10% data set from hit.parquet and do some benchmark from arrow-rs side.
> >
> >
> > That is an interesting idea -- to make make a benchmark in arrow-rs that runs against hits.parquet (and hits_partitioned) directly 🤔 (and e.g. could require downloading those files before running).
>
> I am trying to do the first step, may be we can download a partition hit.parquet, and pick it as the data set to arrow-rs, because we have 100 partition file, it seems about %1 data which can be mocked.
I have been thinking about it too -- I probably won't have a chance this weekend but can work on it next week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org