You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "askoa (via GitHub)" <gi...@apache.org> on 2023/02/08 21:40:05 UTC
[GitHub] [arrow-rs] askoa opened a new pull request, #3679: fix: take_run benchmark parameter
askoa opened a new pull request, #3679:
URL: https://github.com/apache/arrow-rs/pull/3679
# Which issue does this PR close?
# Rationale for this change
I recently noticed an issue with the parameters in the benchmark `primitive_run_take`. The benchmark was not taking indices across the entire logical array but rather constrained to the length of take array. I updated the parameters to take indices across the entire logical array. There is performance regression because more values are taken from the array. I have furnished the performance differences below
<details>
<summary><h4>Benchmark results</h4></summary>
```
primitive_run_take/(run_array_len:512, physical_array_len:64, take_len:512)
time: [23.347 µs 23.508 µs 23.713 µs]
change: [-1.5238% +0.0502% +1.6099%] (p = 0.95 > 0.05)
No change in performance detected.
Found 17 outliers among 100 measurements (17.00%)
3 (3.00%) high mild
14 (14.00%) high severe
primitive_run_take/(run_array_len:512, physical_array_len:128, take_len:512)
time: [23.477 µs 23.616 µs 23.804 µs]
change: [-1.2725% +0.2827% +1.9661%] (p = 0.75 > 0.05)
No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
1 (1.00%) high mild
10 (10.00%) high severe
primitive_run_take/(run_array_len:1024, physical_array_len:256, take_len:512)
time: [23.653 µs 23.778 µs 23.975 µs]
change: [-1.3315% +0.3299% +2.0635%] (p = 0.72 > 0.05)
No change in performance detected.
Found 13 outliers among 100 measurements (13.00%)
3 (3.00%) high mild
10 (10.00%) high severe
primitive_run_take/(run_array_len:1024, physical_array_len:256, take_len:1024)
time: [47.861 µs 48.179 µs 48.621 µs]
change: [-0.3026% +1.4710% +3.2931%] (p = 0.10 > 0.05)
No change in performance detected.
Found 8 outliers among 100 measurements (8.00%)
4 (4.00%) high mild
4 (4.00%) high severe
primitive_run_take/(run_array_len:2048, physical_array_len:512, take_len:512)
time: [24.226 µs 24.287 µs 24.356 µs]
change: [+0.7574% +3.6336% +7.6136%] (p = 0.02 < 0.05)
Change within noise threshold.
Found 11 outliers among 100 measurements (11.00%)
3 (3.00%) high mild
8 (8.00%) high severe
primitive_run_take/(run_array_len:2048, physical_array_len:512, take_len:1024)
time: [53.774 µs 54.093 µs 54.615 µs]
change: [+12.948% +15.613% +18.678%] (p = 0.00 < 0.05)
Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
2 (2.00%) high mild
11 (11.00%) high severe
primitive_run_take/(run_array_len:4096, physical_array_len:1024, take_len:512)
time: [25.297 µs 25.365 µs 25.440 µs]
change: [+4.7053% +7.0111% +9.3052%] (p = 0.00 < 0.05)
Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
5 (5.00%) high mild
8 (8.00%) high severe
primitive_run_take/(run_array_len:4096, physical_array_len:1024, take_len:1024)
time: [59.299 µs 59.752 µs 60.351 µs]
change: [+21.297% +23.389% +25.448%] (p = 0.00 < 0.05)
Performance has regressed.
Found 16 outliers among 100 measurements (16.00%)
8 (8.00%) high mild
8 (8.00%) high severe
```
</details>
# What changes are included in this PR?
# Are there any user-facing changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] ursabot commented on pull request #3679: fix: take_run benchmark parameter
Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #3679:
URL: https://github.com/apache/arrow-rs/pull/3679#issuecomment-1424197325
Benchmark runs are scheduled for baseline = 7cd29d7353369589c18377de4300c44f91a54462 and contender = 98ce68f3229a12e4e057802c4a2cd74170a66c73. 98ce68f3229a12e4e057802c4a2cd74170a66c73 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/9b26c312f17247348590cc49538f167f...4b0755bdc7084baea891eecdf1fdde02/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/44bed87a65b84e7096c49b21a8d4909e...e82acf77c4b0499cae50248c82e3964e/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/24c87750dc1b4fe8a9b38c513db4eed9...25890105d4b741dabea428bedf24d59b/)
[Skipped :warning: Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/1a7ced0582a24b50a0085ce2aac75221...5128d065d22640f79eed3b2c5339be32/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold merged pull request #3679: fix: take_run benchmark parameter
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold merged PR #3679:
URL: https://github.com/apache/arrow-rs/pull/3679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org