You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "gruuya (via GitHub)" <gi...@apache.org> on 2024/03/27 13:24:12 UTC
[PR] Run TPC-H SF10 during PR benchmarks [arrow-datafusion]
gruuya opened a new pull request, #9822:
URL: https://github.com/apache/arrow-datafusion/pull/9822
## Which issue does this PR close?
Progresses #5504.
## Rationale for this change
Make the variance a smaller component in benchmarks by using a larger SF
and thus reduce the noise/false-positives.
## What changes are included in this PR?
Run SF 10 in PR benchmarks too.
## Are these changes tested?
## Are there any user-facing changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run TPC-H SF10 during PR benchmarks [arrow-datafusion]
Posted by "gruuya (via GitHub)" <gi...@apache.org>.
gruuya commented on code in PR #9822:
URL: https://github.com/apache/arrow-datafusion/pull/9822#discussion_r1542010046
##########
.github/workflows/pr_benchmarks.yml:
##########
@@ -64,6 +66,7 @@ jobs:
cd benchmarks
./bench.sh run tpch
+ ./bench.sh run tpch10
Review Comment:
Ok I've added both `tpch_mem10` and `tpch_mem`, so that we can observe and compare the noise level for each one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run TPC-H SF10 during PR benchmarks [arrow-datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #9822:
URL: https://github.com/apache/arrow-datafusion/pull/9822#discussion_r1541274346
##########
.github/workflows/pr_benchmarks.yml:
##########
@@ -64,6 +66,7 @@ jobs:
cd benchmarks
./bench.sh run tpch
+ ./bench.sh run tpch10
Review Comment:
Perhaps we could run `tpch10_mem` as well if it doesn't run OOM, or `tpch_mem` otherwise which should have less variance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run TPC-H SF10 during PR benchmarks [arrow-datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #9822:
URL: https://github.com/apache/arrow-datafusion/pull/9822#discussion_r1541271323
##########
.github/workflows/pr_benchmarks.yml:
##########
@@ -28,8 +28,9 @@ jobs:
cd benchmarks
mkdir data
- # Setup the TPC-H data set with a scale factor of 10
+ # Setup the TPC-H data sets for scale factors 1 and 10
./bench.sh data tpch
Review Comment:
Does it make sense to do SF=1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run TPC-H SF10 during PR benchmarks [arrow-datafusion]
Posted by "gruuya (via GitHub)" <gi...@apache.org>.
gruuya commented on code in PR #9822:
URL: https://github.com/apache/arrow-datafusion/pull/9822#discussion_r1541317743
##########
.github/workflows/pr_benchmarks.yml:
##########
@@ -64,6 +66,7 @@ jobs:
cd benchmarks
./bench.sh run tpch
+ ./bench.sh run tpch10
Review Comment:
Yeah I can add tpch10_mem as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run TPC-H SF10 during PR benchmarks [arrow-datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan merged PR #9822:
URL: https://github.com/apache/arrow-datafusion/pull/9822
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run TPC-H SF10 during PR benchmarks [arrow-datafusion]
Posted by "gruuya (via GitHub)" <gi...@apache.org>.
gruuya commented on code in PR #9822:
URL: https://github.com/apache/arrow-datafusion/pull/9822#discussion_r1541317023
##########
.github/workflows/pr_benchmarks.yml:
##########
@@ -28,8 +28,9 @@ jobs:
cd benchmarks
mkdir data
- # Setup the TPC-H data set with a scale factor of 10
+ # Setup the TPC-H data sets for scale factors 1 and 10
./bench.sh data tpch
Review Comment:
Fair point; I guess it can be beneficial to detect minor systemic/non-linear regressions that are larger than the noise level, but smaller then the sensitivity of SF 10?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org