You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Joe McDonnell (Jira)" <ji...@apache.org> on 2024/02/21 00:25:00 UTC
[jira] [Resolved] (IMPALA-12038) Switch report_benchmark_results.py to Python 3
[ https://issues.apache.org/jira/browse/IMPALA-12038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joe McDonnell resolved IMPALA-12038.
------------------------------------
Fix Version/s: Impala 4.4.0
Assignee: Joe McDonnell
Resolution: Fixed
> Switch report_benchmark_results.py to Python 3
> ----------------------------------------------
>
> Key: IMPALA-12038
> URL: https://issues.apache.org/jira/browse/IMPALA-12038
> Project: IMPALA
> Issue Type: Sub-task
> Components: Infrastructure
> Affects Versions: Impala 4.3.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
> Fix For: Impala 4.4.0
>
>
> report_benchmark_results.py is used by the bin/single_node_perf_run.py script (used by the perf-AB-test Jenkins job). The script is comparing the results stored in two JSON files. In some configurations (e.g. running TPC-DS with many iterations), the JSON files are massive (~4GB). report_benchmark_results.py uses massive amounts of memory and can oversubscribe the machine.
> Python 2 is substantially less efficient than Python 3 around memory usage for this case:
> {noformat}
> Python 2 as-is:
> Memusage: ~30GB, spiking to 43+GB
> real 2m35.975s
> user 2m14.102s
> sys 0m20.922s
> Python 3:
> Memusage: ~8GB, spiking to 10.5GB
> real 2m5.453s
> user 1m55.692s
> sys 0m8.946s{noformat}
> I suspect this may be related to differences in Unicode representation, i.e. [https://peps.python.org/pep-0393/]
> Independent of the larger python 2 to python 3 migration, we should go ahead and migrate this script.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org