You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/04/26 21:37:45 UTC
[GitHub] [arrow-datafusion] alamb opened a new pull request, #6134: Alamb/better compare
alamb opened a new pull request, #6134:
URL: https://github.com/apache/arrow-datafusion/pull/6134
# Which issue does this PR close?
Part of https://github.com/apache/arrow-datafusion/issues/6127
# Rationale for this change
I want to be able to easily understand what is being compared and reported. Currently the reporting from `compare.py` is somewhat opaque:
1. It averages benchmark run timings (which increases the variance)
2. The header columns are non sensical
# What changes are included in this PR?
1. Use `min` of the iteration times to report the benchmark results -- I think this minimizes run to run variance and is the fairest way to asses performance
2. Use the filename as the column header
# Are these changes tested?
No, it is a tool only change
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #6134: Improve `compare.py` output to use `min` times and better column titles
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #6134:
URL: https://github.com/apache/arrow-datafusion/pull/6134
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed pull request #6134: Improve `compare.py` output to use `min` times and better column titles
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #6134: Improve `compare.py` output to use `min` times and better column titles
URL: https://github.com/apache/arrow-datafusion/pull/6134
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] comphead commented on a diff in pull request #6134: Improve `compare.py` output to use `min` times and better column titles
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #6134:
URL: https://github.com/apache/arrow-datafusion/pull/6134#discussion_r1180569668
##########
benchmarks/compare.py:
##########
@@ -64,14 +61,9 @@ def load_from(cls, data: Dict[str, Any]) -> QueryRun:
def execution_time(self) -> float:
assert len(self.iterations) >= 1
- # If we don't have enough samples, median() is probably
- # going to be a worse measure than just an average.
- if len(self.iterations) < MEAN_THRESHOLD:
- method = statistics.mean
- else:
- method = statistics.median
-
- return method(iteration.elapsed for iteration in self.iterations)
+ # Use minimum execution time to account for variations / other
+ # things the system was doing
+ return min(iteration.elapsed for iteration in self.iterations)
Review Comment:
lgtm, but imho, can `min` be deceptive comparing to `avg/mean/median`? 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #6134: Improve `compare.py` output to use `min` times and better column titles
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6134:
URL: https://github.com/apache/arrow-datafusion/pull/6134#discussion_r1180601182
##########
benchmarks/compare.py:
##########
@@ -64,14 +61,9 @@ def load_from(cls, data: Dict[str, Any]) -> QueryRun:
def execution_time(self) -> float:
assert len(self.iterations) >= 1
- # If we don't have enough samples, median() is probably
- # going to be a worse measure than just an average.
- if len(self.iterations) < MEAN_THRESHOLD:
- method = statistics.mean
- else:
- method = statistics.median
-
- return method(iteration.elapsed for iteration in self.iterations)
+ # Use minimum execution time to account for variations / other
+ # things the system was doing
+ return min(iteration.elapsed for iteration in self.iterations)
Review Comment:
I agree it is misleading -- in terms of measuring a change between datafusion versions, I think `min` will give us the least variance between runs and represents best case performance.
However, it doesn't really give a sense for how much variation is across runs 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org