You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2025/02/27 18:44:00 UTC
[jira] [Commented] (IMPALA-13781) report_benchmark_result.py uses the wrong calculation for median diff %
[ https://issues.apache.org/jira/browse/IMPALA-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17931255#comment-17931255 ]
ASF subversion and git services commented on IMPALA-13781:
----------------------------------------------------------
Commit 9033e9aa08dea4f014ffa775a4fd28d721d2eaa3 in impala's branch refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=9033e9aa0 ]
IMPALA-13781: Fix "Median Diff %" for report_benchmark_results.py
The calculation that report_benchmark_results.py uses for
"Median Diff %" is using the wrong divisor. For an AB test,
it should be dividing by the base median time (i.e. A), but
this is actually dividing by the new median time (i.e. B).
That gives the wrong range and can result in "Median Diff %"
values of >100%. This fixes the calculation to divide by the
base median time (i.e. A).
Change-Id: I6f9da14bc7d5b6ef6a795b9024760f8cae77d72f
Reviewed-on: http://gerrit.cloudera.org:8080/22532
Reviewed-by: Riza Suminto <ri...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> report_benchmark_result.py uses the wrong calculation for median diff %
> -----------------------------------------------------------------------
>
> Key: IMPALA-13781
> URL: https://issues.apache.org/jira/browse/IMPALA-13781
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.6.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> The benchmark report includes a column with the "Median Diff(%)", but it is being calculated improperly. It can produce % reductions greater than 100% because it is dividing by the new result rather than the base result:
> {noformat}
> # median uses "results", but it should use "ref_results"
> median = results[SORTED][int(len(results[SORTED]) / 2)]
> all_diffs = [x - y for x in results[SORTED] for y in ref_results[SORTED]]
> all_diffs.sort()
> self.median_diff = all_diffs[int(len(all_diffs) / 2)] / median{noformat}
> In an AB test, the median variable used as the divisor should be the A value (i.e. the base / reference value). Instead, this is using the B value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org