You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2025/08/08 05:30:10 UTC

[Impala-ASF-CR] IMPALA-14294: Fix bin/single node perf run.py to work with compression levels

Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23270


Change subject: IMPALA-14294: Fix bin/single_node_perf_run.py to work with compression levels
......................................................................

IMPALA-14294: Fix bin/single_node_perf_run.py to work with compression levels

bin/single_node_perf_run.py and the supporting scripts currently don't
support compression levels. It is useful to be able to compare compression
levels for these benchmarks, so this adds that support. There are several
parts:
 - Add support for parsing the compression codecs with a level (test_dimension.py)
 - Plumb through the compression level to the compression_codec query option
   (generate-schema-statements.py)
 - Adjust filenames / database names to incorporate the compression level
   to avoid collisions (test_dimension.py, generate-schema-statements.py)
   For example, the database for tpcds parquet/zstd:6 would be
   tpcds42_parquet/zstd_level6. Database names can't contain "-", so this
   specifies negative compression levels with negX. e.g. tpcds42_parquet_zstd_levelneg1
   This keeps the naming the same if the compression level is not specified.
 - Unify the logic around the table format string to simplify the handling
   (report_benchmark_results.py / query.py / test_file_parser.py)

Testing:
 - Ran perf-AB-test with zstd:6, lz4:3, zstd:-1, and without specifying
   compression levels.

Change-Id: If4d3e22f5ac0ff1a41f67e002b72f15523363cc2
---
M testdata/bin/generate-schema-statements.py
M tests/benchmark/report_benchmark_results.py
M tests/common/test_dimensions.py
M tests/performance/query.py
M tests/util/test_file_parser.py
5 files changed, 86 insertions(+), 62 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/23270/1
-- 
To view, visit http://gerrit.cloudera.org:8080/23270
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If4d3e22f5ac0ff1a41f67e002b72f15523363cc2
Gerrit-Change-Number: 23270
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14294: Fix bin/single node perf run.py to work with compression levels

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23270

to look at the new patch set (#2).

Change subject: IMPALA-14294: Fix bin/single_node_perf_run.py to work with compression levels
......................................................................

IMPALA-14294: Fix bin/single_node_perf_run.py to work with compression levels

bin/single_node_perf_run.py and the supporting scripts currently don't
support compression levels. It is useful to be able to compare compression
levels for these benchmarks, so this adds that support. There are several
parts:
 - Add support for parsing the compression codecs with a level (test_dimension.py)
 - Plumb through the compression level to the compression_codec query option
   (generate-schema-statements.py)
 - Adjust filenames / database names to incorporate the compression level
   to avoid collisions (test_dimension.py, generate-schema-statements.py)
   For example, the database for tpcds parquet/zstd:6 would be
   tpcds42_parquet/zstd_level6. Database names can't contain "-", so this
   specifies negative compression levels with negX. e.g. tpcds42_parquet_zstd_levelneg1
   This keeps the naming the same if the compression level is not specified.
 - Unify the logic around the table format string to simplify the handling
   (report_benchmark_results.py / query.py / test_file_parser.py)

Testing:
 - Ran perf-AB-test with zstd:6, lz4:3, zstd:-1, and without specifying
   compression levels.

Change-Id: If4d3e22f5ac0ff1a41f67e002b72f15523363cc2
---
M testdata/bin/generate-schema-statements.py
M tests/benchmark/report_benchmark_results.py
M tests/common/test_dimensions.py
M tests/infra/test_run_workload.py
M tests/performance/query.py
M tests/util/test_file_parser.py
6 files changed, 88 insertions(+), 64 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/23270/2
-- 
To view, visit http://gerrit.cloudera.org:8080/23270
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If4d3e22f5ac0ff1a41f67e002b72f15523363cc2
Gerrit-Change-Number: 23270
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-14294: Fix bin/single node perf run.py to work with compression levels

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23270 )

Change subject: IMPALA-14294: Fix bin/single_node_perf_run.py to work with compression levels
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/20099/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23270
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If4d3e22f5ac0ff1a41f67e002b72f15523363cc2
Gerrit-Change-Number: 23270
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 09 Aug 2025 00:54:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14294: Fix bin/single node perf run.py to work with compression levels

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23270 )

Change subject: IMPALA-14294: Fix bin/single_node_perf_run.py to work with compression levels
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/20093/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23270
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If4d3e22f5ac0ff1a41f67e002b72f15523363cc2
Gerrit-Change-Number: 23270
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 05:58:49 +0000
Gerrit-HasComments: No