[Impala-ASF-CR] IMPALA-15555: Limit the number of profiles preserved by single node perf run.py

["Joe McDonnell (Code Review)" <ge...@cloudera.org>]

Hello Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/25107

to look at the new patch set (#4).

Change subject: IMPALA-15555: Limit the number of profiles preserved by single_node_perf_run.py
......................................................................

IMPALA-15555: Limit the number of profiles preserved by single_node_perf_run.py

Currently, single_node_perf_run.py preserves profiles for every
iteration of a query. When the number of iterations get high,
this is a considerable amount of output. Since single_node_perf_run.py
is used in Jenkins jobs, this can lead to excessive disk space
use from these jobs.

As cleanup, this moves the code that writes the profile files
from bin/single_node_perf_run.py to bin/run-workload.py. This
allows it to operate on the original data structures rather
than reparsing bin/run-workload.py's JSON output.
bin/run-workload.py takes a --profile_dir flag to specify
the output directory for the profiles. It uses the same
profile file names as the old code when num_clients=1.
When num_clients>1, it includes the client number into the
resulting file name to avoid collisions.

This also adds a --max_profiles option to bin/run-workload.py
to limit the number of profiles preserved. This will keep the
first N runs of each query. Keeping the first N runs preserves
the existing behavior for the profile diff file. When there
are multiple clients, keep the first N profiles per query per
client. bin/single_node_perf_run.py adds an equivalent
--max_profiles option that it passes through to
bin/run-workload.py. The JSON output continues to include
every execution, so this continues to be available to
report_benchmark_results.py.

Testing:
 - Added tests in infra/test_run_workload.py

Change-Id: I173121590ee8c5f5169977748ff84a53218cf50c
Assisted-by: Claude Sonnet 5.5 (Claude Code)
---
M bin/run-workload.py
M bin/single_node_perf_run.py
M tests/infra/test_run_workload.py
3 files changed, 133 insertions(+), 40 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/25107/4
-- 
To view, visit http://gerrit.cloudera.org:8080/25107
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I173121590ee8c5f5169977748ff84a53218cf50c
Gerrit-Change-Number: 25107
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>