You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Arnab Karmakar (Code Review)" <ge...@cloudera.org> on 2026/02/07 17:21:47 UTC
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Arnab Karmakar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23950
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
IMPALA-12191: Add hardware and OS details to runtime profile
This patch adds backend hardware (CPU model, core count) and OS
distribution information to the runtime profile. The information
is aggregated across all backends used by a query and displayed
in a concise format.
The aggregation summarizes heterogeneous cluster configurations
by grouping identical CPU/OS configurations and showing their
counts, making it easy to see hardware diversity at a glance.
Testing:
- Added test_backend_hardware_info_in_profile() test
- Manually verified the output in runtime profiles
TODO (for future patches):
- Add cache sizes (L1, L2, L3) information
- Add NUMA node details
- Add memory capacity and type information
- More detailed CPU information (clock speed,
CPU flags like AVX, AVX2, etc.)
Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
---
M be/src/runtime/coordinator.cc
M be/src/scheduling/scheduler.cc
M be/src/service/impala-server.cc
M common/protobuf/admission_control_service.proto
M common/protobuf/statestore_service.proto
M tests/query_test/test_observability.py
6 files changed, 115 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/23950/1
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 8: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 8
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Sun, 03 May 2026 11:53:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 6:
(6 comments)
Sorry to take so long, I appreciate this work.
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/backend-machine-info.h
File be/src/runtime/backend-machine-info.h:
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/backend-machine-info.h@28
PS6, Line 28: /// Wrapper for MachineInfoPB with utility methods for aggregation.
: struct BackendMachineInfo {
Everything that this struct is doing can be done on the original MachineInfoPB struct. Remove this struct and have BackendMachineInfoAggregator work directly on the MachineInfoPB.
If GetCpuConfigString() worked on a MachineInfoPB, we could use it in the Coordinator::BackendState::Init() logic. It probably makes sense as a regular function, but if we use a regular function we'd want to rename it to something like MachineInfoPBToCpuConfigString(). It could also make sense as static method on the BackendMachineInfoAggregator.
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/backend-machine-info.h@34
PS6, Line 34: /// Returns true if this struct has valid machine information.
: bool IsValid() const {
: return !cpu_model_name.empty() && cpu_num_cores > 0 && !os_distribution.empty();
: }
Two things here:
1. I don't want to do a validity check for the structure as a whole. Partial information is still useful. Knowing that something has 32 cores even without knowing the CPU model is still interesting. Knowing the OS distribution is still interesting without knowing the CPU information. The logic for each field should handle its own validity.
2. For the aggregation for the profile, I would rather know about all the nodes, even if that means some of the values are converted to "unknown" or "invalid". Let's say we have a 30 node cluster. If 2 nodes don't report their CPU model, one option is to report the information about 28 nodes and discard those two nodes. The other option is to report about all 30 nodes and report "Unknown" (or something like that) for 2 of them. I'd rather do the second. That way, it all adds up.
In practice, I don't think the APIs ever return values that would trip up this validity check. If it can't detect the CPU model, the CpuInfo will return "unknown". If it can't detect the number of CPUs, it will return 1.
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/backend-machine-info.h@66
PS6, Line 66: if (cpu_model_name.empty() || cpu_num_cores == 0) return "";
As I wrote elsewhere, the algorithm should handle invalid things while still displaying what it can. I don't think we ever have an empty CPU model or num cores = 0, but I'd rather see "Unknown (0 cores)" than an empty string.
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/coordinator-backend-state.cc@94
PS6, Line 94: if (backend_exec_params_.has_machine_info()) {
This is always set, so remove the conditional code (and optionally DCHECK that it is set).
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/coordinator-backend-state.cc@96
PS6, Line 96: if (machine_info.has_os_distribution()) {
I would also remove the conditional logic here and for the cpu model name / cpu num cores.
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/scheduling/scheduler.cc
File be/src/scheduling/scheduler.cc:
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/scheduling/scheduler.cc@1502
PS6, Line 1502: if (be_desc.has_machine_info()) {
If we know that something is always set, we try to avoid having conditional code to handle if it isn't set. A few of the fields immediately above this are "optional" but are always set and we don't check. To be cautious, we would add a DCHECK that something is set. It can often be easier to assert that something is set than handle the possibility that it isn't set in all the different places it is used.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 6
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Apr 2026 23:34:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13419/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 8
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Sun, 03 May 2026 07:23:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello David Rorke, Surya Hebbar, Joe McDonnell, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23950
to look at the new patch set (#3).
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
IMPALA-12191: Add hardware and OS details to runtime profile
This patch adds backend hardware (CPU model, core count) and OS
distribution information to the runtime profile. The information
is displayed in two places:
1. Per-node profiles: Each backend's CPU and OS info appears at
the top of its host profile for easy correlation with performance
2. Aggregated summary: Unique configurations are grouped with counts
Example profile output:
- Per-node profiles (CPU/OS info at top of each host):
Per Node Profiles:
ccycloud.impala-host.root.comops.site:27000:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
- HostCpuSysPercentage (50.000ms): 3, 4, 4, 4
...
ccycloud.impala-host.root.comops.site:27001:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
...
- Aggregated summary (for quick overview):
Per Node Peak Memory Usage: host:27000(400 KB) host:27001(352 KB) host:27002(352 KB)
Per Node Bytes Read: host:27000(161 KB) host:27001(158 KB) host:27002(158 KB)
Backend CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores) (3)
Backend OS Info: Ubuntu 22.04.5 LTS (3)
- Heterogeneous cluster example:
Backend CPU Info: Intel Xeon E5-2680 v4 (28 cores) (15), Intel Xeon E5-2660 v3 (20 cores) (3)
Backend OS Info: CentOS Linux 7 (15), Ubuntu 20.04.3 LTS (3)
Testing:
- Added Python test test_backend_hardware_info_in_profile() that validates
per-node and aggregated hardware info
- Added C++ unit tests for BackendHardwareInfo and
BackendHardwareInfoAggregator covering single/multiple configs, invalid
data handling, and deterministic ordering
- Manually tested with 3-node minicluster
Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
---
M be/src/runtime/CMakeLists.txt
A be/src/runtime/backend-hardware-info-test.cc
A be/src/runtime/backend-hardware-info.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator.cc
M be/src/scheduling/scheduler.cc
M be/src/service/impala-server.cc
M common/protobuf/admission_control_service.proto
M common/protobuf/statestore_service.proto
M tests/query_test/test_observability.py
10 files changed, 508 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/23950/3
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 3
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13169/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 4
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 06:29:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21598/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Sat, 07 Feb 2026 17:48:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 1:
(4 comments)
Thank you for working on this
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG@22
PS1, Line 22: TODO (for future patches):
: - Add cache sizes (L1, L2, L3) information
: - Add NUMA node details
: - Add memory capacity and type information
: - More detailed CPU information (clock speed,
: CPU flags like AVX, AVX2, etc.)
> TODOs maybe better to put in JIRAs or comments within the codebase at appro
I agree with Surya that we don't need to list the future work in detail.
Sometimes, we mention general direction of future work like "In future, this can be extended to other host information like cache sizes, etc." but we have other places to track the actual work items. Sometimes, work is left to a follow-up JIRA and a commit message will mention that JIRA. We use that sparingly. The audience is mainly future readers of the commit message, so it is about what they will find useful.
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/runtime/coordinator.cc@1447
PS1, Line 1447: // TODO(IMPALA-8126): Move to host profiles
> This TODO seems to mention that we should move this to per node profiles if
Let's keep it separate. It isn't really related to what this JIRA is about.
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/runtime/coordinator.cc@1453
PS1, Line 1453: // Add hardware and OS information
> As Joe had mentioned in the JIRA, it's definitely worth aggregating here fo
I'm ok with adding OS / CPU to the per node profile. For future host information, I might want to avoid some of the more verbose things to avoid bloating the profile output.
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/service/impala-server.cc@2678
PS1, Line 2678: be_desc->set_cpu_model_name(CpuInfo::model_name());
: be_desc->set_cpu_num_cores(CpuInfo::num_cores());
: be_desc->set_os_distribution(OsInfo::os_distribution());
Grouping these fields into their own struct would help keep the code clean, especially if we will add more fields:
1. The code in Scheduler::ComputeBackendExecParams() can pass the struct along. Adding a new field wouldn't require that code to change.
2. The code to aggregate the structs could be abstracted out as an aggregator class. Coordinator::ComputeQuerySummary() code can initialize the aggregator class before the loop, call Merge() on each struct as it loops over the backends, then pull information out of the aggregator to put in the profile. (Coordinator::ResourceUtilization does an aggregation like this.) Depending on how we define the API, this may allow the Coordinator::ComputeQuerySummary() code to stay the same as we add fields to the struct.
3. If there is an aggregator class that is somewhat self-contained, then it can be unit tested with hand built structs (including cases with mismatching values).
This may be overkill when it is just a couple fields, but it does make it substantially easier to test.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 25 Feb 2026 01:54:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21931/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 6
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Mar 2026 03:45:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13053/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Sat, 07 Feb 2026 17:26:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 8:
(6 comments)
Thanks for the comments!
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/backend-machine-info.h
File be/src/runtime/backend-machine-info.h:
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/backend-machine-info.h@28
PS6, Line 28:
:
> Everything that this struct is doing can be done on the original MachineInf
Great suggestion. Done.
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/backend-machine-info.h@34
PS6, Line 34:
:
:
:
> Two things here:
Done
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/backend-machine-info.h@66
PS6, Line 66:
> As I wrote elsewhere, the algorithm should handle invalid things while stil
Done
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/coordinator-backend-state.cc@94
PS6, Line 94: // Add machine information to per-node profile
> This is always set, so remove the conditional code (and optionally DCHECK t
Done
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/runtime/coordinator-backend-state.cc@96
PS6, Line 96: const MachineInfoPB& machine_info = backend
> I would also remove the conditional logic here and for the cpu model name /
Done
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/scheduling/scheduler.cc
File be/src/scheduling/scheduler.cc:
http://gerrit.cloudera.org:8080/#/c/23950/6/be/src/scheduling/scheduler.cc@1502
PS6, Line 1502: *backend.second.exec_params->muta
> If we know that something is always set, we try to avoid having conditional
Done
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 8
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Sun, 03 May 2026 07:22:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 3:
(11 comments)
Thanks for the comments.
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG@12
PS1, Line 12: 1. Per-node profiles
> Add the example output in the commit message.
Done
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG@14
PS1, Line 14: 2. Aggregated summary: Unique configurations are grouped with counts
> The minicluster has 3 executors running, so a plan that runs across the 3 e
Done
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG@20
PS1, Line 20: ccycloud.impala-host.root.comops.site:27000:
> You need to update the profiles in testdata/impala-profiles for the impala-
Yes the fields are optional and backward compatible. We dont need any changes here.
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG@22
PS1, Line 22: CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
: - HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
: - HostCpuSysPercentage (50.000ms): 3, 4, 4, 4
: ...
: ccycloud.impala-host.root.comops.site:27001:
: OS Info: Ubuntu 22.04.5 LTS
> I agree with Surya that we don't need to list the future work in detail.
Done
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/runtime/coordinator.cc@1399
PS1, Line 1399: BackendHardwareInfo::FromBackendExecParams(backend_state->exec_params());
> Ah, I see the aggregation here.
Done
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/runtime/coordinator.cc@1447
PS1, Line 1447: }
> Let's keep it separate. It isn't really related to what this JIRA is about.
Done
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/runtime/coordinator.cc@1453
PS1, Line 1453: }
> I'm ok with adding OS / CPU to the per node profile. For future host inform
Done. Added os and cpu info to the per node profile, and yeah its better to keep the per-node profile lean.
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/service/impala-server.cc@2678
PS1, Line 2678: be_desc->set_process_start_time(CurrentTimeString());
: }
: be_desc->set_version(GetBuildVersion(/* compact */ true)
> Grouping these fields into their own struct would help keep the code clean,
Thanks for the thorough explanation, Ive introduced a new struct BackendHardwareInfo and an aggregator class BackendHardwareInfoAggregator. Now new fields can be added cleanly. Ive also added unit tests for the newly added struct and the aggregator class.
http://gerrit.cloudera.org:8080/#/c/23950/1/tests/query_test/test_observability.py
File tests/query_test/test_observability.py:
http://gerrit.cloudera.org:8080/#/c/23950/1/tests/query_test/test_observability.py@1155
PS1, Line 1155: \
> You should print the entire profile in the error message as well.
Done
http://gerrit.cloudera.org:8080/#/c/23950/1/tests/query_test/test_observability.py@1168
PS1, Line 1168:
> Same here.
Done
http://gerrit.cloudera.org:8080/#/c/23950/1/tests/query_test/test_observability.py@1171
PS1, Line 1171: assert os_info_match is not None, \
: "Could not extract Backend OS I
> Write example of where multiple OS and multiple types of CPU info are prese
I wrote examples for it. With my current knowledge, Im not sure how do I test this in such an env. I believe since its working for my current dev env, and the code changes are also made at the right places thanks to Joe's guidance, it should work on heterogenous env too.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 3
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 05:24:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 1: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/runtime/coordinator.cc@1399
PS1, Line 1399: // Aggregate CPU information
Ah, I see the aggregation here.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:03:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 4:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/23950/4/be/src/runtime/backend-hardware-info.h
File be/src/runtime/backend-hardware-info.h:
http://gerrit.cloudera.org:8080/#/c/23950/4/be/src/runtime/backend-hardware-info.h@29
PS4, Line 29: BackendHardwareInfo
Since we include OS settings, can we change this to BackendMachineInfo?
http://gerrit.cloudera.org:8080/#/c/23950/4/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/23950/4/be/src/runtime/coordinator.cc@1442
PS4, Line 1442: // Add aggregated hardware and OS information
: if (hw_info_aggregator.HasData()) {
: string cpu_summary = hw_info_aggregator.GetCpuSummary();
: if (!cpu_summary.empty()) {
: query_profile_->AddInfoString("Backend CPU Info", cpu_summary);
: }
:
: string os_summary = hw_info_aggregator.GetOsSummary();
: if (!os_summary.empty()) {
: query_profile_->AddInfoString("Backend OS Info", os_summary);
: }
: }
This is ok for now. At some point when we have more fields, we may want the BackendHardwareAggregator to return a map with key/value pairs. This would simply add those to the query_profile_ without needing to know about each one.
http://gerrit.cloudera.org:8080/#/c/23950/4/common/protobuf/statestore_service.proto
File common/protobuf/statestore_service.proto:
http://gerrit.cloudera.org:8080/#/c/23950/4/common/protobuf/statestore_service.proto@94
PS4, Line 94: // CPU model name
: optional string cpu_model_name = 16;
:
: // Number of CPU cores available on this backend
: optional int32 cpu_num_cores = 17;
:
: // OS distribution name
: optional string os_distribution = 18;
One small thing: Let's go ahead and turn this into its own struct / message. The constructor for BackendHardwareInfo would take this in rather than the full BackendDescriptorPB. The information can be a single field for BackendExecParamsPB and the code in scheduler can set a single field that won't change when we add more information.
I think statestore_server.proto is the right place to define it, because admission_control_service.proto includes it and can use the struct.
The code for BackendHardwareInfo is basically identical to what this struct would be, so we may not need that struct if we add this one.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 4
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Mar 2026 20:37:24 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 6:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/23950/4/be/src/runtime/backend-hardware-info.h
File be/src/runtime/backend-hardware-info.h:
http://gerrit.cloudera.org:8080/#/c/23950/4/be/src/runtime/backend-hardware-info.h@29
PS4, Line 29:
> Since we include OS settings, can we change this to BackendMachineInfo?
Done
http://gerrit.cloudera.org:8080/#/c/23950/4/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/23950/4/be/src/runtime/coordinator.cc@1442
PS4, Line 1442: // Add aggregated machine information
: if (machine_info_aggregator.HasData()) {
: string cpu_summary = machine_info_aggregator.GetCpuSummary();
: if (!cpu_summary.empty()) {
: query_profile_->AddInfoString("Backend CPU Info", cpu_summary);
: }
:
: string os_summary = machine_info_aggregator.GetOsSummary();
: if (!os_summary.empty()) {
: query_profile_->AddInfoString("Backend OS Info", os_summary);
: }
: }
> This is ok for now. At some point when we have more fields, we may want the
True and since we have only a few fields right now, Im leaving this unchanged.
http://gerrit.cloudera.org:8080/#/c/23950/4/common/protobuf/statestore_service.proto
File common/protobuf/statestore_service.proto:
http://gerrit.cloudera.org:8080/#/c/23950/4/common/protobuf/statestore_service.proto@94
PS4, Line 94: // local time zone.
: optional string process_start_time = 13;
:
: // The pretty-printed string representation of program version and build version for
: // this backend.
: optional string version = 14;
:
: // Unique identifier of an executor w
> One small thing: Let's go ahead and turn this into its own struct / message
Thanks for the tip, I added a new message `MachineInfoPB` and used that in `BackendDescriptorPB`.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 6
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Mar 2026 03:18:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13168/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 3
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 05:26:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG@14
PS1, Line 14: The aggregation summarizes heterogeneous cluster configurations
> Can you provide some examples of this output? Specifically the grouping? I
I only have this output produced by the code from my local dev env:
Backend CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores) (1)
Backend OS Info: Ubuntu 22.04.5 LTS (1)
I havent and dont know how to test this on a heterogenous cluster.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Feb 2026 16:50:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 8:
Hand tested this locally and confirmed the output
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 8
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 May 2026 20:00:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 4
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 10:54:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello David Rorke, Surya Hebbar, Joe McDonnell, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23950
to look at the new patch set (#6).
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
IMPALA-12191: Add hardware and OS details to runtime profile
This patch adds backend hardware (CPU model, core count) and OS
distribution information to the runtime profile. The information
is displayed in two places:
1. Per-node profiles: Each backend's CPU and OS info appears at
the top of its host profile for easy correlation with performance
2. Aggregated summary: Unique configurations are grouped with counts
Example profile output:
- Per-node profiles (CPU/OS info at top of each host):
Per Node Profiles:
ccycloud.impala-host.root.comops.site:27000:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
- HostCpuSysPercentage (50.000ms): 3, 4, 4, 4
...
ccycloud.impala-host.root.comops.site:27001:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
...
- Aggregated summary (for quick overview):
Per Node Peak Memory Usage: host:27000(400 KB) host:27001(352 KB) host:27002(352 KB)
Per Node Bytes Read: host:27000(161 KB) host:27001(158 KB) host:27002(158 KB)
Backend CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores) (3)
Backend OS Info: Ubuntu 22.04.5 LTS (3)
- Heterogeneous cluster example:
Backend CPU Info: Intel Xeon E5-2680 v4 (28 cores) (15), Intel Xeon E5-2660 v3 (20 cores) (3)
Backend OS Info: CentOS Linux 7 (15), Ubuntu 20.04.3 LTS (3)
Testing:
- Added Python test test_backend_hardware_info_in_profile() that validates
per-node and aggregated hardware info
- Added C++ unit tests for BackendHardwareInfo and
BackendHardwareInfoAggregator covering single/multiple configs, invalid
data handling, and deterministic ordering
- Manually tested with 3-node minicluster
Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
---
M be/src/runtime/CMakeLists.txt
A be/src/runtime/backend-machine-info-test.cc
A be/src/runtime/backend-machine-info.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator.cc
M be/src/scheduling/scheduler.cc
M be/src/service/impala-server.cc
M common/protobuf/admission_control_service.proto
M common/protobuf/statestore_service.proto
M tests/query_test/test_observability.py
10 files changed, 525 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/23950/6
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 6
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello David Rorke, Surya Hebbar, Joe McDonnell, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23950
to look at the new patch set (#8).
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
IMPALA-12191: Add hardware and OS details to runtime profile
This patch adds backend hardware (CPU model, core count) and OS
distribution information to the runtime profile. The information
is displayed in two places:
1. Per-node profiles: Each backend's CPU and OS info appears at
the top of its host profile for easy correlation with performance
2. Aggregated summary: Unique configurations are grouped with counts
Example profile output:
- Per-node profiles (CPU/OS info at top of each host):
Per Node Profiles:
ccycloud.impala-host.root.comops.site:27000:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
- HostCpuSysPercentage (50.000ms): 3, 4, 4, 4
...
ccycloud.impala-host.root.comops.site:27001:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
...
- Aggregated summary (for quick overview):
Per Node Peak Memory Usage: host:27000(400 KB) host:27001(352 KB) host:27002(352 KB)
Per Node Bytes Read: host:27000(161 KB) host:27001(158 KB) host:27002(158 KB)
Backend CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores) (3)
Backend OS Info: Ubuntu 22.04.5 LTS (3)
- Heterogeneous cluster example:
Backend CPU Info: Intel Xeon E5-2680 v4 (28 cores) (15), Intel Xeon E5-2660 v3 (20 cores) (3)
Backend OS Info: CentOS Linux 7 (15), Ubuntu 20.04.3 LTS (3)
Testing:
- Added Python test test_backend_hardware_info_in_profile() that validates
per-node and aggregated hardware info
- Added C++ unit tests for BackendHardwareInfo and
BackendHardwareInfoAggregator covering single/multiple configs, invalid
data handling, and deterministic ordering
- Manually tested with 3-node minicluster
Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
---
M be/src/runtime/CMakeLists.txt
A be/src/runtime/backend-machine-info-aggregator-test.cc
A be/src/runtime/backend-machine-info-aggregator.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator.cc
M be/src/scheduling/scheduler.cc
M be/src/service/impala-server.cc
M common/protobuf/admission_control_service.proto
M common/protobuf/statestore_service.proto
M tests/query_test/test_observability.py
10 files changed, 469 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/23950/8
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 8
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 3:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/21822/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 3
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 05:38:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 1:
(8 comments)
The specific CPU and OS info string should be added to the respective per node profile. This is the direct way to identify and corelate performance issues or other problems with OS or CPU, otherwise we cannot quickly identify which OS belongs to which host. Later we can aggregate as we previously did.
It should look something like this...
Per Node Profiles:
surya-Precision-5560:27000:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
Filter 0 arrival: 606ms
Filter 2 arrival: 606ms
- HostCpuIoWaitPercentage (50.000ms): 0...
- HostCpuSysPercentage (50.000ms): 0, 0, 2...
- HostCpuUserPercentage (50.000ms): 2, 2, 2...
surya-Precision-5560:27001:
OS Info: Ubuntu 20.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
Filter 0 arrival: 100ms
Filter 2 arrival: 606ms
- HostCpuIoWaitPercentage (50.000ms): 0...
- HostCpuSysPercentage (50.000ms): 0, 0, 2...
- HostCpuUserPercentage (50.000ms): 2, 2, 2...
surya-Precision-5560:27002:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
Filter 0 arrival: 506ms
Filter 2 arrival: 606ms
- HostCpuIoWaitPercentage (50.000ms): 0...
- HostCpuSysPercentage (50.000ms): 0, 0, 2...
- HostCpuUserPercentage (50.000ms): 2, 2, 2...
Other than this, the patchset looks good.
Thanks for working on it.
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG@12
PS1, Line 12: in a concise format.
Add the example output in the commit message.
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG@20
PS1, Line 20: - Manually verified the output in runtime profiles
You need to update the profiles in testdata/impala-profiles for the impala-profile-tool.
Maybe if the thrift is optional, the output may not change.
In that case, it's okay to leave as it is.
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG@22
PS1, Line 22: TODO (for future patches):
: - Add cache sizes (L1, L2, L3) information
: - Add NUMA node details
: - Add memory capacity and type information
: - More detailed CPU information (clock speed,
: CPU flags like AVX, AVX2, etc.)
TODOs maybe better to put in JIRAs or comments within the codebase at appropriate places, they should not be put in commit messages.
Commit messages are permanent and should not contain TODOs that maybe part of future changes, unless its absolutely necessary.
Please remove this.
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/runtime/coordinator.cc@1447
PS1, Line 1447: // TODO(IMPALA-8126): Move to host profiles
This TODO seems to mention that we should move this to per node profiles if possible.
Please take a look at this, whether this can be done now.
http://gerrit.cloudera.org:8080/#/c/23950/1/be/src/runtime/coordinator.cc@1453
PS1, Line 1453: // Add hardware and OS information
As Joe had mentioned in the JIRA, it's definitely worth aggregating here for easier summary.
But we cannot know which Host/Coordinator/Executor is which OS or CPU version easily. We would need to go to each executor's root page like before and check.
The proper way is to add the specific OS version to each per node profile, and then aggregate here like you are doing now.
http://gerrit.cloudera.org:8080/#/c/23950/1/tests/query_test/test_observability.py
File tests/query_test/test_observability.py:
http://gerrit.cloudera.org:8080/#/c/23950/1/tests/query_test/test_observability.py@1155
PS1, Line 1155: "Could not extract Backend CPU Info from profile"
You should print the entire profile in the error message as well.
Like it was done before...
"Could not extract Backend CPU Info from profile: \n" + profile
http://gerrit.cloudera.org:8080/#/c/23950/1/tests/query_test/test_observability.py@1168
PS1, Line 1168: "Could not extract Backend OS Info from profile"
Same here.
http://gerrit.cloudera.org:8080/#/c/23950/1/tests/query_test/test_observability.py@1171
PS1, Line 1171: # Validate format: Should contain an OS name and a count in parentheses
: # Example: "Ubuntu 22.04.5 LTS (1)"
Write example of where multiple OS and multiple types of CPU info are present as well.
Also see if you can test it out.
If you cannot write a test for it, see if it works as expected in such an environment, or manually simulate such a case through hardcoded cases.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Feb 2026 10:41:16 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
IMPALA-12191: Add hardware and OS details to runtime profile
This patch adds backend hardware (CPU model, core count) and OS
distribution information to the runtime profile. The information
is displayed in two places:
1. Per-node profiles: Each backend's CPU and OS info appears at
the top of its host profile for easy correlation with performance
2. Aggregated summary: Unique configurations are grouped with counts
Example profile output:
- Per-node profiles (CPU/OS info at top of each host):
Per Node Profiles:
ccycloud.impala-host.root.comops.site:27000:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
- HostCpuSysPercentage (50.000ms): 3, 4, 4, 4
...
ccycloud.impala-host.root.comops.site:27001:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
...
- Aggregated summary (for quick overview):
Per Node Peak Memory Usage: host:27000(400 KB) host:27001(352 KB) host:27002(352 KB)
Per Node Bytes Read: host:27000(161 KB) host:27001(158 KB) host:27002(158 KB)
Backend CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores) (3)
Backend OS Info: Ubuntu 22.04.5 LTS (3)
- Heterogeneous cluster example:
Backend CPU Info: Intel Xeon E5-2680 v4 (28 cores) (15), Intel Xeon E5-2660 v3 (20 cores) (3)
Backend OS Info: CentOS Linux 7 (15), Ubuntu 20.04.3 LTS (3)
Testing:
- Added Python test test_backend_hardware_info_in_profile() that validates
per-node and aggregated hardware info
- Added C++ unit tests for BackendHardwareInfo and
BackendHardwareInfoAggregator covering single/multiple configs, invalid
data handling, and deterministic ordering
- Manually tested with 3-node minicluster
Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Reviewed-on: http://gerrit.cloudera.org:8080/23950
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
---
M be/src/runtime/CMakeLists.txt
A be/src/runtime/backend-machine-info-aggregator-test.cc
A be/src/runtime/backend-machine-info-aggregator.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator.cc
M be/src/scheduling/scheduler.cc
M be/src/service/impala-server.cc
M common/protobuf/admission_control_service.proto
M common/protobuf/statestore_service.proto
M tests/query_test/test_observability.py
10 files changed, 469 insertions(+), 0 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Joe McDonnell: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 9
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Surya Hebbar (Code Review)" <ge...@cloudera.org>.
Surya Hebbar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 6:
The change is looking good, though I still need to review the most recent patchset.
Before I do, I wanted to point out a couple of things regarding your Gerrit workflow.
-> Building for each Patchset:
- I noticed that across all your changes, you’ve been running a full parallel-all-tests-ub2004 build to get the "Verified +1" label on every single patchset, even for minimal changes.
- Please keep in mind that this label is not a mandate for every single patchset.
- Although the job is named gerrit-verify-dryrun, it still builds and runs all the core tests. Triggering it for every small change consumes valuable CI resources and hogs the Jenkins queue for others.
- A good rule of thumb: Trigger the full build on your first patchset, or if you make major architectural changes that impact multiple domains you are less familiar with.
- Once you know which tests are impacted, you can just run those specific tests locally for any minor tweaks you make. If the reviewer believes your changes are ready to be committed, they will trigger a final, full build for you.
-> Usage of distinct topic markers:
- A topic marker is generally used to group together multiple commits that belong to the same overarching feature. Please refer to the following documentation.
https://gerrit-review.googlesource.com/Documentation/intro-user.html#topics
- Unless you have multiple commits that need to be grouped together under a single JIRA, using a topic marker for a single standalone commit is redundant.
- As I had mentioned to you on Slack before, for single-commit changes, it is better to push normally without a topic, like most contributors do.
I am not saying you should be overly conservative with your Gerrit contributions. I am just saving you the trouble and cutting through what is not required.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 6
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Mar 2026 11:55:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21823/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 4
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 06:12:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Sat, 07 Feb 2026 22:12:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG@14
PS1, Line 14: The aggregation summarizes heterogeneous cluster configurations
> I only have this output produced by the code from my local dev env:
The minicluster has 3 executors running, so a plan that runs across the 3 executors would have 3 nodes each mentioning their Backend CPU/OS. I would like an excerpt of that profile.
My reading on "grouping identical CPU/OS" would be that this is grouped in the profile somehow. It's not at all clear to me what that would look like or that this patch achieves that goal.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Feb 2026 19:09:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23950/1//COMMIT_MSG@14
PS1, Line 14: The aggregation summarizes heterogeneous cluster configurations
Can you provide some examples of this output? Specifically the grouping? I don't understand how the code would produce this grouping.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Feb 2026 16:13:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Arnab Karmakar has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 1:
I started the minicluster with 3 executors and the profile output produced (with mt_dop=0):
Backend startup latencies: Count: 3, sum: 3ms, min / max: 1ms / 1ms, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms
Slowest backend to start up: ccycloud.power-host.root.comops.site:27000
Per Node Peak Memory Usage: ccycloud.power-host.root.comops.site:27000(400.00 KB) ccycloud.power-host.root.comops.site:27001(352.00 KB) ccycloud.power-host.root.comops.site:27002(352.00 KB)
Per Node Bytes Read: ccycloud.power-host.root.comops.site:27000(161.09 KB) ccycloud.power-host.root.comops.site:27001(158.70 KB) ccycloud.power-host.root.comops.site:27002(158.66 KB)
Per Node User Time: ccycloud.power-host.root.comops.site:27000(4.386ms) ccycloud.power-host.root.comops.site:27001(3.364ms) ccycloud.power-host.root.comops.site:27002(2.074ms)
Per Node System Time: ccycloud.power-host.root.comops.site:27000(897.000us) ccycloud.power-host.root.comops.site:27001(0.000ns) ccycloud.power-host.root.comops.site:27002(1.595ms)
Backend CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores) (3)
Backend OS Info: Ubuntu 22.04.5 LTS (3)
- ExchangeScanRatio: 0.00
- FiltersReceived: 0 (0)
How it works:
1. Each backend (executor or coordinator) reports its hardware specs via BackendDescriptorPB. Information is collected when each backend registers with the statestore.
2. During query execution, the coordinator collects info from all participating backends
3. Identical configurations are grouped together with a count
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 1
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 17:41:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 8: Code-Review+2
Thank you, this looks great!
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 8
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 May 2026 19:54:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22296/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 8
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Sun, 03 May 2026 07:43:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13227/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 6
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Mar 2026 03:21:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Arnab Karmakar (Code Review)" <ge...@cloudera.org>.
Hello David Rorke, Surya Hebbar, Joe McDonnell, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23950
to look at the new patch set (#4).
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
IMPALA-12191: Add hardware and OS details to runtime profile
This patch adds backend hardware (CPU model, core count) and OS
distribution information to the runtime profile. The information
is displayed in two places:
1. Per-node profiles: Each backend's CPU and OS info appears at
the top of its host profile for easy correlation with performance
2. Aggregated summary: Unique configurations are grouped with counts
Example profile output:
- Per-node profiles (CPU/OS info at top of each host):
Per Node Profiles:
ccycloud.impala-host.root.comops.site:27000:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
- HostCpuSysPercentage (50.000ms): 3, 4, 4, 4
...
ccycloud.impala-host.root.comops.site:27001:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
...
- Aggregated summary (for quick overview):
Per Node Peak Memory Usage: host:27000(400 KB) host:27001(352 KB) host:27002(352 KB)
Per Node Bytes Read: host:27000(161 KB) host:27001(158 KB) host:27002(158 KB)
Backend CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores) (3)
Backend OS Info: Ubuntu 22.04.5 LTS (3)
- Heterogeneous cluster example:
Backend CPU Info: Intel Xeon E5-2680 v4 (28 cores) (15), Intel Xeon E5-2660 v3 (20 cores) (3)
Backend OS Info: CentOS Linux 7 (15), Ubuntu 20.04.3 LTS (3)
Testing:
- Added Python test test_backend_hardware_info_in_profile() that validates
per-node and aggregated hardware info
- Added C++ unit tests for BackendHardwareInfo and
BackendHardwareInfoAggregator covering single/multiple configs, invalid
data handling, and deterministic ordering
- Manually tested with 3-node minicluster
Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
---
M be/src/runtime/CMakeLists.txt
A be/src/runtime/backend-hardware-info-test.cc
A be/src/runtime/backend-hardware-info.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator.cc
M be/src/scheduling/scheduler.cc
M be/src/service/impala-server.cc
M common/protobuf/admission_control_service.proto
M common/protobuf/statestore_service.proto
M tests/query_test/test_observability.py
10 files changed, 503 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/23950/4
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 4
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13168/
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 3
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 08:02:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12191: Add hardware and OS details to runtime profile
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23950 )
Change subject: IMPALA-12191: Add hardware and OS details to runtime profile
......................................................................
Patch Set 6: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Gerrit-Change-Number: 23950
Gerrit-PatchSet: 6
Gerrit-Owner: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: Arnab Karmakar <ar...@gmail.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Mar 2026 07:40:33 +0000
Gerrit-HasComments: No