You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org> on 2025/10/23 20:08:21 UTC
[Impala-ASF-CR] IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
Mihaly Szjatinya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23583
Change subject: IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
......................................................................
IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
This continues the effort in IMPALA-6070 and consequent tickets.
Still mostly WIP to agree on basic concepts, some features not
finalized / for debugging purposes. Tested locally and on Jenkins,
mostly for individual test suites.
1. Container Infrastructure
- Cgroup v1/v2 Compatibility: Added smart detection for both cgroup
v1 and v2 systems
- Support for cgroup v2 unified hierarchy
(/sys/fs/cgroup/cgroup.controllers)
- Enhanced container startup logging and diagnostic information
- Fixed IMPALA_BUILD_THREADS detection for BE_TEST
- Docker attach readiness improvements with timing optimizations
- CPU workload integration hooks for performance testing
- Better error reporting and container lifecycle management
2. Improved Monitoring and Timeline UI
Timeline Improvements
- 6-graph comprehensive timeline: CPU/Memory/Disk usage for both
Container and Host
- Enhanced JSON data structure for rich metrics collection
- Timeline generation in case of job failure
- Collapsible test results per Test Suite
- Added min_memory_usage_gb and max_memory_usage_gb attributes to
ContainerMonitor
CPU Workload Testing System
- Build and test phase CPU load simulation for realistic metrics
- cpu_workloads.sh: Configurable CPU stress testing utilities for
performance benchmarking
3. Docker Registry Support
- Complete registry push/pull automation
- Human-readable tagging system (repo:tag format instead of
digest-only)
- Docker registry authentication with robot account support
- SHA256 digest handling for image validation and integrity
- Multi-format registry support (standard Docker + Harbor-style
registries)
4. Python 2.7 Compatibility
[Perhaps not needed in the future]
- Replace f-string syntax with .format() method calls for string
formatting
- Replace subprocess.DEVNULL with open(os.devnull, 'w') for
compatibility
- Remove Python 3+ only subprocess.TimeoutExpired exception handling
- Fix various import and syntax compatibility issues
- Ensure all Docker scripts work across Python 2.7+ environments
Dependencies and Integration
- Docker registry with authentication support
TODO:
- Test concurrent runs / tune for better performance
- Fix individual tests
- Auto-mode for most optimal test run
- Finalize test report in Timeline
- Incremental build support
Stats
For tests suites when run separately:
[impala-docker-ubuntu2004-ec2-m6i-8xlarge-ondemand:impala]
SUITE | Result
---------------------|--------------------
BE_TEST | ✅ PASS
| 44m
---------------------|--------------------
FE_TEST | ❌ 4 failures - jwtAuth related
| 1h 26m
---------------------|--------------------
EE_TEST_PARALLEL | ❌ 2 failures
| 1h
---------------------|--------------------
EE_TEST_SERIAL (2) | ✅ PASS
| 47m
---------------------|--------------------
CLUSTER_TEST (2) | ❌ 35 failures - ssl/ipv6 related
| 2h 27m
---------------------|--------------------
CLUSTER_TEST (1) | ✅ PASS
| 3h 9m
---------------------|--------------------
JDBC_TEST | ✅ PASS
build image - 1h5m
docker push - 36min (23GB)
docker pull - 5min
Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
---
A docker/cpu_workloads.sh
M docker/entrypoint.sh
M docker/monitor.py
M docker/test-with-docker.py
M docker/timeline.html.template
5 files changed, 1,330 insertions(+), 88 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/23583/1
--
To view, visit http://gerrit.cloudera.org:8080/23583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
Gerrit-Change-Number: 23583
Gerrit-PatchSet: 1
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
[Impala-ASF-CR] IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23583 )
Change subject: IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20764/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
Gerrit-Change-Number: 23583
Gerrit-PatchSet: 2
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Comment-Date: Thu, 23 Oct 2025 20:36:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/23583 )
Change subject: IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
......................................................................
IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
This continues the effort in IMPALA-6070 and consequent tickets.
Still mostly WIP to agree on basic concepts, some features not
finalized / for debugging purposes. Tested locally and on Jenkins,
mostly for individual test suites.
1. Container Infrastructure
- Cgroup v1/v2 Compatibility: Added smart detection for both cgroup
v1 and v2 systems
- Support for cgroup v2 unified hierarchy
(/sys/fs/cgroup/cgroup.controllers)
- Enhanced container startup logging and diagnostic information
- Fixed IMPALA_BUILD_THREADS detection for BE_TEST
- Docker attach readiness improvements with timing optimizations
- CPU workload integration hooks for performance testing
- Better error reporting and container lifecycle management
2. Improved Monitoring and Timeline UI
Timeline Improvements
- 6-graph comprehensive timeline: CPU/Memory/Disk usage for both
Container and Host
- Enhanced JSON data structure for rich metrics collection
- Timeline generation in case of job failure
- Collapsible test results per Test Suite
- Added min_memory_usage_gb and max_memory_usage_gb attributes to
ContainerMonitor
CPU Workload Testing System
- Build and test phase CPU load simulation for realistic metrics
- cpu_workloads.sh: Configurable CPU stress testing utilities for
performance benchmarking
3. Docker Registry Support
- Complete registry push/pull automation
- Human-readable tagging system (repo:tag format instead of
digest-only)
- Docker registry authentication with robot account support
- SHA256 digest handling for image validation and integrity
- Multi-format registry support (standard Docker + Harbor-style
registries)
4. Python 2.7 Compatibility
[Perhaps not needed in the future]
- Replace f-string syntax with .format() method calls for string
formatting
- Replace subprocess.DEVNULL with open(os.devnull, 'w') for
compatibility
- Remove Python 3+ only subprocess.TimeoutExpired exception handling
- Fix various import and syntax compatibility issues
- Ensure all Docker scripts work across Python 2.7+ environments
Dependencies and Integration
- Docker registry with authentication support
TODO:
- Test concurrent runs / tune for better performance
- Fix individual tests
- Auto-mode for most optimal test run
- Finalize test report in Timeline
- Incremental build support
Stats
For tests suites when run separately:
SUITE | Result
---------------------|--------------------
BE_TEST | ✅ PASS
| 44m
---------------------|--------------------
FE_TEST | ❌ 4 failures - jwtAuth related
| 1h 26m
---------------------|--------------------
EE_TEST_PARALLEL | ❌ 2 failures
| 1h
---------------------|--------------------
EE_TEST_SERIAL (2) | ✅ PASS
| 47m
---------------------|--------------------
CLUSTER_TEST (2) | ❌ 35 failures - ssl/ipv6 related
| 2h 27m
---------------------|--------------------
CLUSTER_TEST (1) | ✅ PASS
| 3h 9m
---------------------|--------------------
JDBC_TEST | ✅ PASS
build image - 1h5m
docker push - 36min (23GB)
docker pull - 5min
Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
---
A docker/cpu_workloads.sh
M docker/entrypoint.sh
M docker/monitor.py
M docker/test-with-docker.py
M docker/timeline.html.template
5 files changed, 1,339 insertions(+), 88 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/23583/3
--
To view, visit http://gerrit.cloudera.org:8080/23583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
Gerrit-Change-Number: 23583
Gerrit-PatchSet: 3
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
[Impala-ASF-CR] IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has restored this change. ( http://gerrit.cloudera.org:8080/23583 )
Change subject: IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
......................................................................
Restored
--
To view, visit http://gerrit.cloudera.org:8080/23583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: restore
Gerrit-Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
Gerrit-Change-Number: 23583
Gerrit-PatchSet: 2
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
[Impala-ASF-CR] IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/23583 )
Change subject: IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
......................................................................
IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
This continues the effort in IMPALA-6070 and consequent tickets.
Still mostly WIP to agree on basic concepts, some features not
finalized / for debugging purposes. Tested locally and on Jenkins,
mostly for individual test suites.
1. Container Infrastructure
- Cgroup v1/v2 Compatibility: Added smart detection for both cgroup
v1 and v2 systems
- Support for cgroup v2 unified hierarchy
(/sys/fs/cgroup/cgroup.controllers)
- Enhanced container startup logging and diagnostic information
- Fixed IMPALA_BUILD_THREADS detection for BE_TEST
- Docker attach readiness improvements with timing optimizations
- CPU workload integration hooks for performance testing
- Better error reporting and container lifecycle management
2. Improved Monitoring and Timeline UI
Timeline Improvements
- 6-graph comprehensive timeline: CPU/Memory/Disk usage for both
Container and Host
- Enhanced JSON data structure for rich metrics collection
- Timeline generation in case of job failure
- Collapsible test results per Test Suite
- Added min_memory_usage_gb and max_memory_usage_gb attributes to
ContainerMonitor
CPU Workload Testing System
- Build and test phase CPU load simulation for realistic metrics
- cpu_workloads.sh: Configurable CPU stress testing utilities for
performance benchmarking
3. Docker Registry Support
- Complete registry push/pull automation
- Human-readable tagging system (repo:tag format instead of
digest-only)
- Docker registry authentication with robot account support
- SHA256 digest handling for image validation and integrity
- Multi-format registry support (standard Docker + Harbor-style
registries)
4. Python 2.7 Compatibility
[Perhaps not needed in the future]
- Replace f-string syntax with .format() method calls for string
formatting
- Replace subprocess.DEVNULL with open(os.devnull, 'w') for
compatibility
- Remove Python 3+ only subprocess.TimeoutExpired exception handling
- Fix various import and syntax compatibility issues
- Ensure all Docker scripts work across Python 2.7+ environments
Dependencies and Integration
- Docker registry with authentication support
TODO:
- Test concurrent runs / tune for better performance
- Fix individual tests
- Auto-mode for most optimal test run
- Finalize test report in Timeline
- Incremental build support
Stats
For tests suites when run separately:
SUITE | Result
---------------------|--------------------
BE_TEST | ✅ PASS
| 44m
---------------------|--------------------
FE_TEST | ❌ 4 failures - jwtAuth related
| 1h 26m
---------------------|--------------------
EE_TEST_PARALLEL | ❌ 2 failures
| 1h
---------------------|--------------------
EE_TEST_SERIAL (2) | ✅ PASS
| 47m
---------------------|--------------------
CLUSTER_TEST (2) | ❌ 35 failures - ssl/ipv6 related
| 2h 27m
---------------------|--------------------
CLUSTER_TEST (1) | ✅ PASS
| 3h 9m
---------------------|--------------------
JDBC_TEST | ✅ PASS
build image - 1h5m
docker push - 36min (23GB)
docker pull - 5min
Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
---
A docker/cpu_workloads.sh
M docker/entrypoint.sh
M docker/monitor.py
M docker/test-with-docker.py
M docker/timeline.html.template
5 files changed, 1,330 insertions(+), 88 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/23583/2
--
To view, visit http://gerrit.cloudera.org:8080/23583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
Gerrit-Change-Number: 23583
Gerrit-PatchSet: 2
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
[Impala-ASF-CR] IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23583 )
Change subject: IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20875/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
Gerrit-Change-Number: 23583
Gerrit-PatchSet: 3
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Nov 2025 17:40:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23583 )
Change subject: IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
......................................................................
Patch Set 3:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23583/3/docker/test-with-docker.py
File docker/test-with-docker.py:
http://gerrit.cloudera.org:8080/#/c/23583/3/docker/test-with-docker.py@41
PS3, Line 41: def _load_registry_config():
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/23583/3/docker/test-with-docker.py@48
PS3, Line 48: _registry_config = _load_registry_config()
flake8: E305 expected 2 blank lines after class or function definition, found 1
--
To view, visit http://gerrit.cloudera.org:8080/23583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
Gerrit-Change-Number: 23583
Gerrit-PatchSet: 3
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Nov 2025 17:19:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
Posted by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org>.
Mihaly Szjatinya has abandoned this change. ( http://gerrit.cloudera.org:8080/23583 )
Change subject: IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
......................................................................
Abandoned
--
To view, visit http://gerrit.cloudera.org:8080/23583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
Gerrit-Change-Number: 23583
Gerrit-PatchSet: 2
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
[Impala-ASF-CR] IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23583 )
Change subject: IMPALA-14477: [WIP] Speed up test execution (Part 2) (Part 1)
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20763/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23583
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I99cc0b944d896d8a86b52ebe255f56086fba5182
Gerrit-Change-Number: 23583
Gerrit-PatchSet: 1
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Comment-Date: Thu, 23 Oct 2025 20:35:35 +0000
Gerrit-HasComments: No