You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "gaurav singh (Code Review)" <ge...@cloudera.org> on 2025/05/20 17:59:26 UTC
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
gaurav singh has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22924
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
IMPALA-13235: Update the build toolchain to include
the latest OpenTelemetry C++ sdk.
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M bin/impala-config.sh
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/1
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 1
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
Patch Set 71:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/22924/68//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22924/68//COMMIT_MSG@39
PS68, Line 39:
: Once the query has closed, the root span is closed.
> In that case, the current child and root spans are ended. No further child
Ack
http://gerrit.cloudera.org:8080/#/c/22924/71//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22924/71//COMMIT_MSG@11
PS71, Line 11: Each Impala query is represented as its own individual OpenTelemetry
: trace.
What happen in retried query and query with child queries (ie., COMPUTE STATS)?
Are they represented as different OTel trace? Please clarify in commit message.
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.h
File be/src/service/client-request-state.h:
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.h@551
PS68, Line 551: otel_trace
> I've never been a fan of the "is_" prefix. To me, it does not add clarity.
Done
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.h@551
PS68, Line 551: nst { return otel_span_manager_.get() != null
> That definitely is the pattern. I didn't find any other instances of casti
Done
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.cc@128
PS68, Line 128: // initialize OpenTelemetry for this query
: VLOG(2) << "Initializing OpenTelemetry for query " << PrintId(query_id());
: otel_span_manager_ = build_span_manager(this);
: otel_span_manager_->StartChildSpanInit();
: }
:
: bool is_external_fe = session_type() == TSessionType::EXTERNAL_FRONTEND;
: // "Impala Backend Timeline" was specifically chosen to exploit the lexicographical
: // ordering defined by the underlying std::map holding the timelines
> sql_stmt is always trimmed. I both tested it and confirmed in the code.
Done
http://gerrit.cloudera.org:8080/#/c/22924/71/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/22924/71/be/src/service/client-request-state.cc@1213
PS71, Line 1213: // TODO: remove this
: // otel_span_manager_->EndChildSpanPlanning();
: // otel_span_manager_.reset();
Remove this?
http://gerrit.cloudera.org:8080/#/c/22924/71/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/22924/71/be/src/service/impala-server.cc@1383
PS71, Line 1383: (*query_handle)->otel_span_manager()->EndChildSpanSubmitted();
Why this is far away from MarkEvent("Query submitted");?
Should span open/close closer to existing MarkEvent?
http://gerrit.cloudera.org:8080/#/c/22924/68/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/68/tests/custom_cluster/test_otel_trace.py@114
PS68, Line 114: # Retrieve the query id and runtime profile from the UI since the query execute call
: # only returns a HiveS
> It's not possible (that I know of) to get the runtime profile when using se
Ack
http://gerrit.cloudera.org:8080/#/c/22924/68/tests/custom_cluster/test_otel_trace.py@249
PS68, Line 249: ERIES": True})
> Unfortunately no. When a query is retried, a second query is launched. Th
Ack
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 71
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Aug 2025 22:46:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
Patch Set 71:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20054/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 71
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Aug 2025 22:50:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "gaurav singh (Code Review)" <ge...@cloudera.org>.
gaurav singh has uploaded a new patch set (#68). ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Adds representation of Impala queries using OpenTelemetry traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. These traces consist of a root span and several child spans.
Each child span has the root as its parent. No child span has another
child span as its parent. Each child span represents one high-level
query lifecycle stage. Each child span also has span attributes that
further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of be "DML', "DDL", or "QUERY".
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
22 files changed, 2,545 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/68
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 68
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
Patch Set 68:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20047/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 68
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Aug 2025 20:47:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 88:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20107/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 88
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sun, 10 Aug 2025 02:43:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 97: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 97
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 22:58:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 1:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19251/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 1
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 May 2025 18:05:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 8:
(7 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel-flags.cc
File be/src/observe/otel-flags.cc:
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel-flags.cc@147
PS8, Line 147: DEFINE_string(otel_tls_minimum_version, "", "String containing the minimum allowed TLS version, "
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel-flags.cc@171
PS8, Line 171: DEFINE_double(otel_retry_policy_initial_backoff_s, 1.0, "Initial backoff delay between retry "
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel-flags.cc@174
PS8, Line 174: DEFINE_validator(otel_retry_policy_initial_backoff_s, [](const char* flagname, double value) {
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel-flags.cc@188
PS8, Line 188: DEFINE_validator(otel_retry_policy_backoff_multiplier, [](const char* flagname, double value) {
line too long (95 > 90)
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel.cc@60
PS8, Line 60:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel.cc@77
PS8, Line 77:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel.cc@87
PS8, Line 87: auto processor = std::unique_ptr<SpanProcessor>(new SimpleSpanProcessor(std::move(exporter)));
line too long (98 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 8
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Jun 2025 23:26:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 5:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19398/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 5
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jun 2025 18:53:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 75:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/22924/71//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22924/71//COMMIT_MSG@11
PS71, Line 11:
: Each I
> What happen in retried query and query with child queries (ie., COMPUTE STA
Compute stats will be handled in another patch. Retried queries are an exception to this statement. Commit msg updated.
http://gerrit.cloudera.org:8080/#/c/22924/72/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/72/be/src/observe/otel.cc@124
PS72, Line 124: return boost::algorithm::istarts_with(sql, "select ");
deleted
http://gerrit.cloudera.org:8080/#/c/22924/71/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/22924/71/be/src/service/client-request-state.cc@1213
PS71, Line 1213: // Like the other Exec(), fill out as much profile information as we're able to.
: summary_profile_->AddInfoString("Query Type", PrintValue(TStmtType::DDL));
: RETURN_IF_ERROR(frontend_->Exec
> Remove this?
Done
http://gerrit.cloudera.org:8080/#/c/22924/71/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/22924/71/be/src/service/impala-server.cc@1383
PS71, Line 1383: return Status(ErrorMsg(TErrorCode::MAX_STATEMENT_LENGTH_EXCEED
> Why this is far away from MarkEvent("Query submitted");?
Done
http://gerrit.cloudera.org:8080/#/c/22924/72/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/72/tests/custom_cluster/test_otel_trace.py@298
PS72, Line 298:
> flake8: W291 trailing whitespace
Done
http://gerrit.cloudera.org:8080/#/c/22924/72/tests/custom_cluster/test_otel_trace.py@298
PS72, Line 298: # Assert Close span.
> line has trailing whitespace
Done
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 75
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Aug 2025 22:19:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 83:
(16 comments)
http://gerrit.cloudera.org:8080/#/c/22924/80/be/src/observe/otel-flags-trace.cc
File be/src/observe/otel-flags-trace.cc:
http://gerrit.cloudera.org:8080/#/c/22924/80/be/src/observe/otel-flags-trace.cc@85
PS80, Line 85: delimiter
> nit. delimiter
Done
http://gerrit.cloudera.org:8080/#/c/22924/80/be/src/observe/otel-flags-trace.cc@159
PS80, Line 159: e
> nit. '
Done
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/observe/otel.cc@243
PS77, Line 243: } else {
: exporter = init_exporter_file();
: }
: VLOG(2) << "OpenTelemetry exporter: " << FLAGS_otel_trace_exporter;
:
> Seems better to check whether exporter is nullptr after init, and do some a
The function called in init_exporter_file() on line 228 does always returns an initialized unique_ptr. Thus, checking for nullptr isn't necessary.
http://gerrit.cloudera.org:8080/#/c/22924/80/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/80/be/src/observe/otel.cc@209
PS80, Line 209:
> should be OtlpFileExporter?
Done
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/observe/span-manager.h
File be/src/observe/span-manager.h:
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/observe/span-manager.h@80
PS79, Line 80:
> nit. behavior
Done
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/observe/span-manager.h@104
PS79, Line 104: y::trace
> nit. deactivate
Done
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/observe/span-manager.h@150
PS79, Line 150: ptr)
> nit. its
Done
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/service/impala-server.cc@1332
PS77, Line 1332: return status;
> nit. unnecessary change
Done
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/service/impala-server.cc@1511
PS77, Line 1511: }
> nit. unnecessary change
Done
http://gerrit.cloudera.org:8080/#/c/22924/80/tests/common/custom_cluster_test_suite.py
File tests/common/custom_cluster_test_suite.py:
http://gerrit.cloudera.org:8080/#/c/22924/80/tests/common/custom_cluster_test_suite.py@682
PS80, Line 682: valid
> nit. valid
Done
http://gerrit.cloudera.org:8080/#/c/22924/80/tests/common/custom_cluster_test_suite.py@692
PS80, Line 692: whose
> nit. whose
Done
http://gerrit.cloudera.org:8080/#/c/22924/80/tests/common/custom_cluster_test_suite.py@722
PS80, Line 722: functio
> nit. function
Done
http://gerrit.cloudera.org:8080/#/c/22924/80/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/80/tests/custom_cluster/test_otel_trace.py@415
PS80, Line 415: log line
> nit. line
Done
http://gerrit.cloudera.org:8080/#/c/22924/79/tests/util/otel_trace.py
File tests/util/otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/79/tests/util/otel_trace.py@35
PS79, Line 35: Additionall
> nit. Additionally
Done
http://gerrit.cloudera.org:8080/#/c/22924/79/tests/util/otel_trace.py@117
PS79, Line 117: extracte
> nit. extracted
Done
http://gerrit.cloudera.org:8080/#/c/22924/79/tests/util/query_profile_util.py
File tests/util/query_profile_util.py:
http://gerrit.cloudera.org:8080/#/c/22924/79/tests/util/query_profile_util.py@82
PS79, Line 82: was
> The sentence not finished, not retried?
Done
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 83
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 17:51:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 85: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 85
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 21:25:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 87:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20104/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 87
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sat, 09 Aug 2025 15:10:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 96: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22924/93/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/93/be/src/observe/span-manager.cc@299
PS93, Line 299:
: if (client_request_state_->exec_request().stmt_type == TStmtType::
> Good point. I'm dropping this attribute and will revisit later.
The backend code for getting the counter value could also be dropped.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 96
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 21:49:17 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#94) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,839 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/94
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 94
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "gaurav singh (Code Review)" <ge...@cloudera.org>.
gaurav singh has restored this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Restored
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: restore
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 4
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "gaurav singh (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22924
to look at the new patch set (#5).
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
IMPALA-13235: Update the build toolchain to include
the latest OpenTelemetry C++ sdk.
Add coordinator trace
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/src/runtime/coordinator.cc
M bin/impala-config.sh
2 files changed, 12 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/5
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 5
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 16:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19493/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 16
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sun, 15 Jun 2025 03:26:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 13:
(4 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22924/13/be/src/observe/otel-flags.cc
File be/src/observe/otel-flags.cc:
http://gerrit.cloudera.org:8080/#/c/22924/13/be/src/observe/otel-flags.cc@208
PS13, Line 208: LOG(ERROR) << "Flag '" << flagname << "' contains an invalid header (missing '='): "
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/22924/13/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/13/be/src/observe/otel.cc@102
PS13, Line 102: chrono::duration<float>(FLAGS_otel_retry_policy_initial_backoff_s);
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22924/13/be/src/observe/otel.cc@138
PS13, Line 138: FLAGS_tls_ciphersuites : FLAGS_otel_tls_cipher_suites;
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22924/13/be/src/observe/otel.cc@153
PS13, Line 153: unique_ptr<SpanExporter> exporter = unique_ptr<SpanExporter>(new OtlpHttpExporter(opts));
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 13
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Jun 2025 19:10:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "gaurav singh (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Jason Fehr, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22924
to look at the new patch set (#13).
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
IMPALA-13235: Update the build toolchain to include
the latest OpenTelemetry C++ sdk.
Add coordinator trace
Add traces for query timeline
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/common/init.cc
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags.cc
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/query-driver.cc
M be/src/runtime/query-driver.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/service/client-request-state.cc
M be/src/service/impala-server.cc
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
19 files changed, 973 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/13
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 13
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
Patch Set 72:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22924/72/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/72/tests/custom_cluster/test_otel_trace.py@298
PS72, Line 298:
flake8: W291 trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22924/72/tests/custom_cluster/test_otel_trace.py@298
PS72, Line 298: span_err_msg = query_status
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 72
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Aug 2025 00:42:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#93) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,837 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/93
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 93
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 96:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20124/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 96
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 20:01:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 95:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22924/95/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/95/be/src/observe/span-manager.cc@388
PS95, Line 388: ATTR_ELAPSED_TIME, std::move(additional_attributes), trace::SpanKind::kInternal, root_);
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/22924/95/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/95/tests/custom_cluster/test_otel_trace.py@26
PS95, Line 26: from tests.util.query_profile_util import parse_db_user, parse_session_id, parse_sql, \
flake8: F401 'tests.util.query_profile_util.parse_num_rows_fetched_from_cache' imported but unused
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 95
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 19:32:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Reviewed-on: http://gerrit.cloudera.org:8080/22924
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,821 insertions(+), 15 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 99
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#81) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M CMakeLists.txt
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,824 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/81
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 81
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 89:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22924/89/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/89/tests/custom_cluster/test_otel_trace.py@33
PS89, Line 33: class TestOtelTrace(CustomClusterTestSuite):
Would be good to also tests which capture traces in abnormal exits like query expired (EXEC_TIME_LIMIT_S exceeded), cancelled.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 89
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 14:48:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
Patch Set 73:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20057/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 73
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Aug 2025 01:16:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 93:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/22924/93/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/93/be/src/observe/span-manager.cc@155
PS93, Line 155: event event
nit: event
http://gerrit.cloudera.org:8080/#/c/22924/93/be/src/observe/span-manager.cc@299
PS93, Line 299: attrs.emplace(ATTR_NUM_ROWS_FETCHED_FROM_CACHE,
: client_request_state_->num_rows_fetched_from_cache_counter());
This is the cached query results from the current query and doesn't survive beyond the query close. Not sure if the intention of this attribute is to capture this? For instance this is not the same as Hive's query result cache. Not sure if we want to capture this in the trace, as this could easily get confusing?
http://gerrit.cloudera.org:8080/#/c/22924/93/be/src/observe/span-manager.cc@393
PS93, Line 393: auto a
Use reference and avoid creating local copies.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 93
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 14:29:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 89:
(16 comments)
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/otel-flags-trace.cc
File be/src/observe/otel-flags-trace.cc:
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/otel-flags-trace.cc@219
PS85, Line 219: DEFINE_int32(otel_trace_batch_queue_size, batch_opts.max_queue_size, "The maximum "
: "buffer/queue size. After the size is reached, spans are dropped. Applicable when "
: "'otel_trace_span_processor' is 'batch'.");
: DEFINE_validator(otel_trace_batch_queue_size, ge_one);
:
: DEFINE_int32(otel_trace_batch_schedule_delay_ms, batch_opts.schedule_delay_millis.count(),
: "The delay interval in milliseconds between two consecutive batch exports. "
: "Applicable when 'otel_trace_span_processor' is 'batch'.");
: DEFINE_validator(otel_trace_batch_schedule_delay_ms, ge_one);
:
: DEFINE_int32(otel_trace_batch_max_batch_size, batch_opts.max_export_batch_size,
: "The maximum batch size of every export to OTel Collector. Applicable when "
: "'otel_trace_span_processor' is 'batch'.");
: DEFINE_validator(otel_trace_batch_max_batch_size, ge_one);
> The span processors are implemented by the opentelemetry-cpp sdk. The defi
Ack.
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/otel.cc@124
PS85, Line 124: return boost::algorithm::istarts_with(sql, "select ");
> for now yes since CTEs can be used in insert dmls (which are not supported
Ack.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.h
File be/src/observe/span-manager.h:
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.h@82
PS87, Line 82: // This behavior is different from the other child spans since the Close span will be
> Not quite, the other StartChildSpan functions will fail a DCHECK. Updated
Ack.
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/span-manager.cc@54
PS85, Line 54: static constexpr char const* ATTR_ERROR_MESSAGE = "ErrorMessage";
: static constexpr char const* ATTR_ERROR_MSG = "ErrorMsg";
> One is an attribute on the root span and the other is an attribute on the c
Ack. Maybe add comment here to help differentiate these two.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@138
PS87, Line 138: DCHECK(client_request_state_ != nullptr) << "Cannot end root span without a valid "
> ASAN build revealed this issue. It is fixed.
Ack.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@144
PS87, Line 144: root_->SetAttribute(ATTR_QUERY_TYPE,
> The query type does not get determined until planning has finished.
It looks like we're still setting this in destructor for the root span, but in End Child Span planning function we're setting the attributes for the child span. One of the things, I still don'g quite understand is what's the difference between attributes in the child span and root span? Is it mainly based on attributes that are invariants or not?
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@159
PS87, Line 159: if (UNLIKELY(client_request_state_->WasRetried())) {
> Retried queries get their own span manager. This matches the behavior of t
Ack.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@180
PS87, Line 180: LOG(INFO) << strings::Substitute("Submitted OpenTelemetry trace with trace_id=\"$0\" "
> Good question, will clarify the wording and move it to the end of the funct
Ack.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@231
PS87, Line 231: {ATTR_ORIGINAL_QUERY_ID, (client_request_state_->IsRetriedQuery() ?
> On the second query of the retry, client_request_state_->IsRetriedQuery() w
Ack.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@363
PS87, Line 363: additional_attributes.insert_or_assign(ATTR_RUNNING, running);
> It's a boolean set to true when the query is actually executing (after plan
Ack.
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.h
File be/src/observe/timed-span.h:
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.h@58
PS89, Line 58: const std::string& start_time_attribute_name,
: const std::string& duration_attribute_name,
> I agree it's not the best design. This is another case where root spans ha
Ack. We just need to ensure we're not passing any local string copies as the key.
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.cc
File be/src/observe/timed-span.cc:
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.cc@99
PS89, Line 99: // Helper function to convert an OpenTelemetry ID to a string representation.
: template <size_t N>
: static string id_to_string(const nostd::span<const uint8_t, N> id) {
: DCHECK(N == 8 || N == 16) << "id_to_string only supports 8 or 16 byte IDs";
:
: ostringstream oss;
: for (auto byte : id) {
: oss << hex << setw(2) << setfill('0') << static_cast<int>(byte);
: }
:
: return oss.str();
: } // function id_to_string
:
: string TimedSpan::SpanId() const {
: if (LIKELY(span_)) {
: return id_to_string(span_->GetContext().span_id().Id());
: }
:
: return "";
: } // function SpanId
:
: string TimedSpan::TraceId() const {
: if (LIKELY(span_)) {
: return id_to_string(span_->GetContext().trace_id().Id());
: }
:
: return "";
: }
> Done, stored them as string but had to return const std::string& since the
Ack.
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/runtime/coordinator.cc@1497
PS84, Line 1497: vector<NetworkAddressPB> host_addrs;
> Yeah, it should be past tense. fixed.
Ack.
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/scheduling/local-admission-control-client.h
File be/src/scheduling/local-admission-control-client.h:
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/scheduling/local-admission-control-client.h@55
PS84, Line 55: bool was_queued_;
> It's used when ending the admission control child span -- https://gerrit.cl
Ack.
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/service/client-request-state.cc@1206
PS84, Line 1206: otel_span_manager_->AddChildSpanEvent("QueryUnregistered");
> past tense, and I think "QueryUnregistered" sounds better
Ack.
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/service/impala-server.cc@3610
PS84, Line 3610: if (FLAGS_is_coordinator && otel_trace_enabled()) {
> good catch! fixed
Ack.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 89
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 14:44:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22924/3/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/22924/3/bin/impala-config.sh@84
PS3, Line 84: export IMPALA_TOOLCHAIN_BUILD_ID_AARCH64=91-f9f56a8a24
Is any update necessary now? It looks like another commit was already built from the same hash.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 3
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 May 2025 22:00:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 8:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/22924/8//COMMIT_MSG
Commit Message:
PS8:
Is there any verifiable constraints once a trace is added into the coordinator? If so, does it make sense to include a minimum smoke test scenarios for that?
http://gerrit.cloudera.org:8080/#/c/22924/8//COMMIT_MSG@10
PS8, Line 10: Add coordinator trace
Probably, that's done in some other patch, but since I don't see it in the gerrit patch, I'd rather ask right here :)
Is it expected to be a notion of context at all as a higher level umbrella for the added coordinator's trace? I'm curious what the OTEL's Context entity is going to be mapped into in Impala.
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel-flags.cc
File be/src/observe/otel-flags.cc:
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel-flags.cc@37
PS8, Line 37: rfind
nit: it's a bit strange to see rfind() but not find() if trying to make sure the string starts with the specified suffix. Is this an attempt to do some extra validation on the string?
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel-flags.cc@43
PS8, Line 43: R"(^(https?://)" // http(s)://
Why to have this if the string was already validated for the presence of such suffixes above? Maybe, it make sense to drop the duplicate validation at lines 37-40 if this regex validator is present?
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel-flags.cc@163
PS8, Line 163: DEFINE_validator(otel_retry_policy_max_attempts, [](const char* flagname, int32_t value) {
: if (value < 0) {
: LOG(ERROR) << "Flag '" << flagname << "' must be greater than or equal to 0.";
: return false;
: }
: return true;
: });
Is it possible to use uint32 flag for otel_retry_policy_max_attempts instead of adding this validator?
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel.cc@67
PS8, Line 67: opts.ssl_min_tls = FLAGS_otel_tls_minimum_version.empty() ?
: "1.2" : FLAGS_otel_tls_minimum_version;
Does this make sense to to set opts.ssl_min_tls at all if --otel_tls_minimum_version is empty or we want to enforce having 1.2 set as minimum all the time?
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel.cc@70
PS8, Line 70: opts.ssl_cipher = FLAGS_otel_ssl_ciphers.empty() ? "" : FLAGS_otel_ssl_ciphers;
Could this be reduced to
opts.ssl_cipher = FLAGS_otel_ssl_ciphers;
?
http://gerrit.cloudera.org:8080/#/c/22924/8/be/src/observe/otel.cc@95
PS8, Line 95: provider
nit: does it make sense to use std::move here as well?
http://gerrit.cloudera.org:8080/#/c/22924/8/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/22924/8/bin/impala-config.sh@210
PS8, Line 210: 1.20.0
They released 1.21.0 just end of May 2025: picking https://github.com/open-telemetry/opentelemetry-cpp/releases/tag/v1.21.0 If it makes sense, consider pickup up the most recent version unless there is some specific reason of not doing so.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 8
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Jun 2025 22:56:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 84:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/22924/84/tests/common/custom_cluster_test_suite.py
File tests/common/custom_cluster_test_suite.py:
http://gerrit.cloudera.org:8080/#/c/22924/84/tests/common/custom_cluster_test_suite.py@692
PS84, Line 692: service: Impala coordinator whose UI will be checked. If None, uses the first
: impalad in the cluster.
no service in the param, maybe can remove?
http://gerrit.cloudera.org:8080/#/c/22924/84/tests/common/custom_cluster_test_suite.py@716
PS84, Line 716: break
unreachable break
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 84
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 19:16:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#83) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,822 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/83
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 83
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 84:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/observe/otel.cc@243
PS77, Line 243: } else {
: exporter = init_exporter_file();
: }
: VLOG(2) << "OpenTelemetry exporter: " << FLAGS_otel_trace_exporter;
:
> The function called in init_exporter_file() on line 228 does always returns
Are we sure that OtlpFileExporterFactory::Create() always returns a non-nullptr unique_ptr? Same to init_exporter_http(). If so, it is good, maybe we add a DCHECK(exporter != nullptr)?
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 84
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 18:24:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 84:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/runtime/coordinator.cc@155
PS84, Line 155: // TODO: trace a query
Can we remove this TODO?
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/runtime/coordinator.cc@1497
PS84, Line 1497: "ReleaseAdmissionControlResources");
Since all the other events are in the past tense, should we call this one "ReleasedAdmissionControlResources" or "AdmissionControlResourcesReleased" ?
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/scheduling/local-admission-control-client.h
File be/src/scheduling/local-admission-control-client.h:
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/scheduling/local-admission-control-client.h@55
PS84, Line 55: bool was_queued_;
why do we need to persist this state in the client? I did not see it being used for any future events?
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/service/client-request-state.cc@1206
PS84, Line 1206: otel_span_manager_->AddChildSpanEvent("UnregisterQuery");
Should this be "UnregisteredQuery" or "QueryUnregistered" ?
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/service/impala-server.cc@3610
PS84, Line 3610: if (otel_trace_enabled()) {
shutdown condition needs to be same as initialization:
if (FLAGS_is_coordinator && otel_trace_enabled())
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 84
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 19:10:06 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 77:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20086/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 77
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 01:05:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 78:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20088/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 78
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 03:42:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 89:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12297/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 89
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sun, 10 Aug 2025 02:21:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 89:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20108/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 89
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sun, 10 Aug 2025 02:48:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 87:
(12 comments)
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/otel-flags-trace.cc
File be/src/observe/otel-flags-trace.cc:
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/otel-flags-trace.cc@219
PS85, Line 219: DEFINE_int32(otel_trace_batch_queue_size, batch_opts.max_queue_size, "The maximum "
: "buffer/queue size. After the size is reached, spans are dropped. Applicable when "
: "'otel_trace_span_processor' is 'batch'.");
: DEFINE_validator(otel_trace_batch_queue_size, ge_one);
:
: DEFINE_int32(otel_trace_batch_schedule_delay_ms, batch_opts.schedule_delay_millis.count(),
: "The delay interval in milliseconds between two consecutive batch exports. "
: "Applicable when 'otel_trace_span_processor' is 'batch'.");
: DEFINE_validator(otel_trace_batch_schedule_delay_ms, ge_one);
:
: DEFINE_int32(otel_trace_batch_max_batch_size, batch_opts.max_export_batch_size,
: "The maximum batch size of every export to OTel Collector. Applicable when "
: "'otel_trace_span_processor' is 'batch'.");
: DEFINE_validator(otel_trace_batch_max_batch_size, ge_one);
What's the difference between batch and simple span processor? Is this documented somewhere?
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/otel.cc@124
PS85, Line 124: return boost::algorithm::istarts_with(sql, "select ");
We're skipping some CTEs also. Is that intentional?
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.h
File be/src/observe/span-manager.h:
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.h@47
PS87, Line 47: }; // enum class ChildSpanType
unnecessary comment
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.h@51
PS87, Line 51: // enum ChildSpanType
unnecessary comment
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.h@82
PS87, Line 82: // This behavior is different from the other child spans since the Close span will be
From the above comments (L73) the behavior seems same as other ChildSpans?
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/span-manager.cc@54
PS85, Line 54: static constexpr char const* ATTR_ERROR_MESSAGE = "ErrorMessage";
: static constexpr char const* ATTR_ERROR_MSG = "ErrorMsg";
What's the difference between these two?
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@138
PS87, Line 138: DCHECK(client_request_state_ != nullptr) << "Cannot end root span without a valid "
We probably need to make sure that we cleanup SpanManager before client_request_state_ gets cleaned up.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@144
PS87, Line 144: root_->SetAttribute(ATTR_QUERY_TYPE,
Why do we need to set this in destructor and not in constructor? Is it because exec_request() may not be setup? The other option is to set this in the root span when Starting/Ending Exec/Plan child spans.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@159
PS87, Line 159: if (UNLIKELY(client_request_state_->WasRetried())) {
Are we using same SpanManager for retried query or a different one?
Retried queries go through admission control and so do we create Spans for admission control and query execution again?
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@180
PS87, Line 180: LOG(INFO) << strings::Substitute("Submitted OpenTelemetry trace with trace_id=\"$0\" "
What are we logging here? Do we flush root span in root_->End()?
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@231
PS87, Line 231: {ATTR_ORIGINAL_QUERY_ID, (client_request_state_->IsRetriedQuery() ?
Query won't be retried until it is past admission and actually executing or starting to execute.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@363
PS87, Line 363: additional_attributes.insert_or_assign(ATTR_RUNNING, running);
It's not clear what <ATTR_RUNNING> = running means since ChildSpanBuilder is called for different stages of query.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 87
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sun, 10 Aug 2025 15:25:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 91:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20114/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 91
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sun, 10 Aug 2025 22:47:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "gaurav singh (Code Review)" <ge...@cloudera.org>.
gaurav singh has abandoned this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Abandoned
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 4
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "gaurav singh (Code Review)" <ge...@cloudera.org>.
Hello Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22924
to look at the new patch set (#2).
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
IMPALA-13235: Update the build toolchain to include
the latest OpenTelemetry C++ sdk.
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M bin/impala-config.sh
1 file changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/2
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 2
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
Patch Set 68:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/22924/68//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22924/68//COMMIT_MSG@39
PS68, Line 39:
: Once the query has closed, the root span is closed.
What happen if query terminate abnormally or cancelled?
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.h
File be/src/service/client-request-state.h:
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.h@551
PS68, Line 551: return static_cast<bool>(otel_span_manager_);
Isn't it better this way?
return otel_span_manager_.get() != nullptr;
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.h@551
PS68, Line 551: otel_trace
nit: is_otel_enabled sounds better.
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.cc@128
PS68, Line 128: && !boost::algorithm::istarts_with(sql_stmt().c_str(), ":")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "copy ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "describe ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "explain ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "load ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "set ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "show ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "use ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "values ")
Create a function for this? Is there any better way to detect the kind of query to trace besides of matching the SQL prefix?
Is sql_stmt always trimmed?
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.cc@1288
PS68, Line 1288: otel_span_manager_->EndChildSpanQueryExecution();
: otel_span_manager()->StartChildSpanClose();
Consistently use otel_span_manager_ everywhere.
http://gerrit.cloudera.org:8080/#/c/22924/68/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/68/tests/custom_cluster/test_otel_trace.py@114
PS68, Line 114: query_id, profile = self.query_id_from_ui(section="completed_queries",
: match_query=query)
Is it not possible to collect profile through self.client.get_runtime_profile() ?
http://gerrit.cloudera.org:8080/#/c/22924/68/tests/custom_cluster/test_otel_trace.py@249
PS68, Line 249: self.query_profile_from_ui(result.query_id)
Is it not possible to collect profile through self.client.get_runtime_profile() ?
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 68
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Aug 2025 21:09:36 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
Patch Set 72:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20056/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 72
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Aug 2025 01:06:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#16) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
IMPALA-13235: Update the build toolchain to include
the latest OpenTelemetry C++ sdk.
Add coordinator trace
Add traces for query timeline
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M CMakeLists.txt
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/runtime/query-driver.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/service/client-request-state.cc
M be/src/service/impala-server.cc
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
20 files changed, 1,190 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/16
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 16
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 76:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20083/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 76
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Aug 2025 22:47:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#80) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M CMakeLists.txt
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,814 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/80
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 80
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 79: Code-Review+1
(5 comments)
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/otel.cc@213
PS76, Line 213: static unique
> Done. It was a leftover from a previous design.
Done
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/span-manager.cc@290
PS76, Line 290: li
> The design originally called for including only the queued reason. After r
Done
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/runtime/coordinator.cc@612
PS79, Line 612:
Done
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/service/client-request-state.cc@789
PS79, Line 789: UpdateCatalogFinished
Done
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/service/client-request-state.cc@1266
PS79, Line 1266: Rows available"
nit: RowsAvailable
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 79
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 03:59:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 81:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20091/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 81
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 05:37:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 85: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 85
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sat, 09 Aug 2025 01:04:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#7) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
IMPALA-13235: Update the build toolchain to include
the latest OpenTelemetry C++ sdk.
Add coordinator trace
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/runtime/coordinator.cc
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
5 files changed, 42 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/7
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 7
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#97) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,821 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/97
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 97
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 98:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12304/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 98
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 22:58:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 97:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20126/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 97
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 23:21:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "gaurav singh (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22924
to look at the new patch set (#6).
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
IMPALA-13235: Update the build toolchain to include
the latest OpenTelemetry C++ sdk.
Add coordinator trace
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/src/runtime/coordinator.cc
1 file changed, 10 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/6
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 6
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19409/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 7
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Jun 2025 19:15:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#71) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Adds representation of Impala queries using OpenTelemetry traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. These traces consist of a root span and several child spans.
Each child span has the root as its parent. No child span has another
child span as its parent. Each child span represents one high-level
query lifecycle stage. Each child span also has span attributes that
further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of be "DML', "DDL", or "QUERY".
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
22 files changed, 2,561 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/71
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 71
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#82) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,826 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/82
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 82
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 82:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12287/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 82
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 17:10:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 82:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20094/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 82
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 17:36:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#86) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,810 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/86
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 86
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 89:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.h
File be/src/observe/timed-span.h:
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.h@38
PS89, Line 38: typedef std::unordered_map<std::string, opentelemetry::common::AttributeValue>
We should use string_view as the key since the keys are always constant string literals. We also avoid unnecessary copy for every key. We're copying multiple strings per query and we should avoid that.
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.h@58
PS89, Line 58: const std::string& start_time_attribute_name,
: const std::string& duration_attribute_name,
We're doing unnecessary copy here. We should not pass these as param but directly access the string literals in the code i.e., ATTR_QUERY_START_TIME and ATTR_ELAPSED_TIME.
We probably should move all the string literals in span-manager.cc to timed-span.h so that they are accessible in other translation units.
//timed-span.h
class TimedSpan {
public:
inline constexpr char const* ATTR_ADM_RESULT = "AdmissionResult";
.....
.....
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.cc
File be/src/observe/timed-span.cc:
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.cc@81
PS89, Line 81: const string& key
we should use string_view here.
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.cc@86
PS89, Line 86: const string& key
use string_view here
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.cc@99
PS89, Line 99: // Helper function to convert an OpenTelemetry ID to a string representation.
: template <size_t N>
: static string id_to_string(const nostd::span<const uint8_t, N> id) {
: DCHECK(N == 8 || N == 16) << "id_to_string only supports 8 or 16 byte IDs";
:
: ostringstream oss;
: for (auto byte : id) {
: oss << hex << setw(2) << setfill('0') << static_cast<int>(byte);
: }
:
: return oss.str();
: } // function id_to_string
:
: string TimedSpan::SpanId() const {
: if (LIKELY(span_)) {
: return id_to_string(span_->GetContext().span_id().Id());
: }
:
: return "";
: } // function SpanId
:
: string TimedSpan::TraceId() const {
: if (LIKELY(span_)) {
: return id_to_string(span_->GetContext().trace_id().Id());
: }
:
: return "";
: }
If SpanId() and TraceId() are called multiple times for the same span object we could consider storing them as class members so we compute them once.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 89
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sun, 10 Aug 2025 20:24:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#90) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,815 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/90
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 90
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 92:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20115/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 92
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sun, 10 Aug 2025 23:54:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 13:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19468/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 13
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Jun 2025 19:27:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#73) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Adds representation of Impala queries using OpenTelemetry traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. These traces consist of a root span and several child spans.
Each child span has the root as its parent. No child span has another
child span as its parent. Each child span represents one high-level
query lifecycle stage. Each child span also has span attributes that
further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of be "DML', "DDL", or "QUERY".
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
22 files changed, 2,643 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/73
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 73
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "gaurav singh (Code Review)" <ge...@cloudera.org>.
Hello Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22924
to look at the new patch set (#3).
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
IMPALA-13235: Update the build toolchain to include
the latest OpenTelemetry C++ sdk.
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M bin/impala-config.sh
1 file changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/3
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 3
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
Patch Set 70:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20048/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 70
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Aug 2025 21:56:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#70) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Adds representation of Impala queries using OpenTelemetry traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. These traces consist of a root span and several child spans.
Each child span has the root as its parent. No child span has another
child span as its parent. Each child span represents one high-level
query lifecycle stage. Each child span also has span attributes that
further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of be "DML', "DDL", or "QUERY".
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
22 files changed, 2,552 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/70
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 70
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#78) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M CMakeLists.txt
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,798 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/78
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 78
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 95:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20123/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 95
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 19:58:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 97:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22924/93/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/93/be/src/observe/span-manager.cc@299
PS93, Line 299:
: if (client_request_state_->exec_request().stmt_type == TStmtType::
> The backend code for getting the counter value could also be dropped.
Done
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 97
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 22:56:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 98: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 98
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Aug 2025 04:11:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 80:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20090/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 80
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 04:38:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 83:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20096/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 83
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 18:16:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 74:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20065/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 74
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Aug 2025 22:35:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 76:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22924/76/tests/util/otel_trace.py
File tests/util/otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/76/tests/util/otel_trace.py@59
PS76, Line 59: def __str__(self):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/22924/76/tests/util/otel_trace.py@67
PS76, Line 67: class OtelTrace():
flake8: E302 expected 2 blank lines, found 1
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 76
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Aug 2025 22:31:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 86:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12292/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 86
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sat, 09 Aug 2025 04:14:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 89: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 89
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sun, 10 Aug 2025 08:23:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 86:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20102/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 86
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sat, 09 Aug 2025 04:42:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#8) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
IMPALA-13235: Update the build toolchain to include
the latest OpenTelemetry C++ sdk.
Add coordinator trace
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/common/init.cc
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags.cc
A be/src/observe/otel.cc
A be/src/observe/otel.h
M be/src/runtime/coordinator.cc
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
10 files changed, 400 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/8
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 8
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "gaurav singh (Code Review)" <ge...@cloudera.org>.
gaurav singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22924/2/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/22924/2/bin/impala-config.sh@90
PS2, Line 90: ${IMPALA_TOOLCHAIN_COMMIT_HASH-f9f56a8a2414dd75c636175af6da930c3f6d6934}
> We probably could infer this from ${IMPALA_TOOLCHAIN_BUILD_ID_X86_64#*-}, w
Done
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 3
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 May 2025 18:58:37 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22924/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/22924/1/bin/impala-config.sh@84
PS1, Line 84: export IMPALA_TOOLCHAIN_BUILD_ID_AARCH64=78-49a326be6f
We also need an updated aarch64 build, and IMPALA_TOOLCHAIN_COMMIT_HASH also needs to be updated to match.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 1
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 20 May 2025 18:01:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 2:
(1 comment)
> Patch Set 2:
>
> Build Failed
>
> https://jenkins.impala.io/job/gerrit-code-review-checks/19279/ : Initial code review checks failed. See linked job for details on the failure.
You'll need to rebase to pick up IMPALA-14067.
http://gerrit.cloudera.org:8080/#/c/22924/2/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/22924/2/bin/impala-config.sh@90
PS2, Line 90: ${IMPALA_TOOLCHAIN_COMMIT_HASH-49a326be6f1836127b1189cea71ea9832b31b7aa}
> We should also update this to match, f9f56a8a24 (or the full hash).
We probably could infer this from ${IMPALA_TOOLCHAIN_BUILD_ID_X86_64#*-}, which we test below.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 2
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 May 2025 18:43:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
Patch Set 70:
> Uploaded patch set 70.
This patch only fixes clang tidy warnings.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 70
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Aug 2025 21:28:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has removed a vote on this change.
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 60
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 87: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 87
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sat, 09 Aug 2025 20:22:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 75:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22924/75/tests/util/otel_trace.py
File tests/util/otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/75/tests/util/otel_trace.py@59
PS75, Line 59: def __str__(self):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/22924/75/tests/util/otel_trace.py@67
PS75, Line 67: class OtelTrace():
flake8: E302 expected 2 blank lines, found 1
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 75
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Aug 2025 22:19:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 75:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20066/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 75
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Aug 2025 22:46:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#77) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M CMakeLists.txt
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,800 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/77
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 77
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 93:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20116/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 93
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 02:30:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 94:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/span-manager.cc@54
PS85, Line 54: static constexpr char const* ATTR_STATE = "State";
:
> Ack. Maybe add comment here to help differentiate these two.
I reorganized all of these strings separating them into relevant sections.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@144
PS87, Line 144: debug_log_span(root_.get(), "Root", query_id_, true);
> It looks like we're still setting this in destructor for the root span, but
This particular attribute is being set on the Planning child span in DoEndChildSpanPlanning and on the root span in EndChildSpanClose. It's not being set in the dtor anymore because that was referencing the out-of-scope client_request_state_.
It's a historical vs real-time monitoring use case difference. Child span attributes are used to monitor queries while they are executing while root span attributes are used to surface information about queries after they are completed (to avoid having to iterate down into child spans).
There are some duplications between attributes on the root and child spans. The reason is that queries could take a very long time to finish. The root span does not get sent to the trace visualization system until the query completes. Child spans are sent as soon as they finish. Thus, to enable real time monitoring of queries while they are executing, attributes appear on child spans when they applicable.
http://gerrit.cloudera.org:8080/#/c/22924/93/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/93/be/src/observe/span-manager.cc@155
PS93, Line 155:
> nit: event
Done
http://gerrit.cloudera.org:8080/#/c/22924/93/be/src/observe/span-manager.cc@299
PS93, Line 299:
: if (client_request_state_->exec_request().stmt_type == TStmtType::
> This is the cached query results from the current query and doesn't survive
Good point. I'm dropping this attribute and will revisit later.
http://gerrit.cloudera.org:8080/#/c/22924/93/be/src/observe/span-manager.cc@393
PS93, Line 393:
> Use reference and avoid creating local copies.
Done
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.h
File be/src/observe/timed-span.h:
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.h@58
PS89, Line 58: OtelAttributesMap&& attributes,
: opentelemetry::trace::SpanKind span_kind =
> Ack. We just need to ensure we're not passing any local string copies as th
Double checked, we are only passing constexpr const char*
http://gerrit.cloudera.org:8080/#/c/22924/89/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/89/tests/custom_cluster/test_otel_trace.py@33
PS89, Line 33: class TestOtelTrace(CustomClusterTestSuite):
> Would be good to also tests which capture traces in abnormal exits like que
Agreed. Added to the list of test cases to implement.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 94
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 19:27:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#96) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,839 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/96
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 96
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 84:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12288/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 84
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 17:51:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 16:
(3 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22924/16/be/src/observe/otel-flags.cc
File be/src/observe/otel-flags.cc:
http://gerrit.cloudera.org:8080/#/c/22924/16/be/src/observe/otel-flags.cc@194
PS16, Line 194: LOG(ERROR) << "Flag '" << flagname << "' contains an invalid header (missing '='): "
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/22924/16/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/16/be/src/observe/otel.cc@113
PS16, Line 113: chrono::duration<float>(FLAGS_otel_retry_policy_initial_backoff_s);
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/22924/16/be/src/observe/otel.cc@149
PS16, Line 149: FLAGS_tls_ciphersuites : FLAGS_otel_tls_cipher_suites;
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 16
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sun, 15 Jun 2025 03:08:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has removed Alexey Serbin from this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Removed reviewer Alexey Serbin.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 18
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22924/2/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/22924/2/bin/impala-config.sh@90
PS2, Line 90: ${IMPALA_TOOLCHAIN_COMMIT_HASH-49a326be6f1836127b1189cea71ea9832b31b7aa}
We should also update this to match, f9f56a8a24 (or the full hash).
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 2
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 May 2025 17:50:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#72) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Adds representation of Impala queries using OpenTelemetry traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. These traces consist of a root span and several child spans.
Each child span has the root as its parent. No child span has another
child span as its parent. Each child span represents one high-level
query lifecycle stage. Each child span also has span attributes that
further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of be "DML', "DDL", or "QUERY".
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
22 files changed, 2,674 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/72
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 72
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#88) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,817 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/88
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 88
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 90:
(17 comments)
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/otel-flags-trace.cc
File be/src/observe/otel-flags-trace.cc:
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/otel-flags-trace.cc@219
PS85, Line 219: DEFINE_int32(otel_trace_batch_queue_size, batch_opts.max_queue_size, "The maximum "
: "buffer/queue size. After the size is reached, spans are dropped. Applicable when "
: "'otel_trace_span_processor' is 'batch'.");
: DEFINE_validator(otel_trace_batch_queue_size, ge_one);
:
: DEFINE_int32(otel_trace_batch_schedule_delay_ms, batch_opts.schedule_delay_millis.count(),
: "The delay interval in milliseconds between two consecutive batch exports. "
: "Applicable when 'otel_trace_span_processor' is 'batch'.");
: DEFINE_validator(otel_trace_batch_schedule_delay_ms, ge_one);
:
: DEFINE_int32(otel_trace_batch_max_batch_size, batch_opts.max_export_batch_size,
: "The maximum batch size of every export to OTel Collector. Applicable when "
: "'otel_trace_span_processor' is 'batch'.");
: DEFINE_validator(otel_trace_batch_max_batch_size, ge_one);
> What's the difference between batch and simple span processor? Is this docu
The span processors are implemented by the opentelemetry-cpp sdk. The definitions of each can be found there.
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/otel.cc@124
PS85, Line 124: return boost::algorithm::istarts_with(sql, "select ");
> We're skipping some CTEs also. Is that intentional?
for now yes since CTEs can be used in insert dmls (which are not supported with this patch)
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.h
File be/src/observe/span-manager.h:
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.h@47
PS87, Line 47: };
> unnecessary comment
Done
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.h@51
PS87, Line 51:
> unnecessary comment
Done
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.h@82
PS87, Line 82: // other start child span functions since the Close span will be started in error
> From the above comments (L73) the behavior seems same as other ChildSpans?
Not quite, the other StartChildSpan functions will fail a DCHECK. Updated comments to clarify the differences.
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/85/be/src/observe/span-manager.cc@54
PS85, Line 54: static constexpr char const* ATTR_ERROR_MESSAGE = "ErrorMessage";
: static constexpr char const* ATTR_ERROR_MSG = "ErrorMsg";
> What's the difference between these two?
One is an attribute on the root span and the other is an attribute on the child spans.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@138
PS87, Line 138: root_->End();
> We probably need to make sure that we cleanup SpanManager before client_req
ASAN build revealed this issue. It is fixed.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@144
PS87, Line 144: scope_.reset();
> Why do we need to set this in destructor and not in constructor? Is it beca
The query type does not get determined until planning has finished.
Moved this to the end child span planning function too.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@159
PS87, Line 159: } else {
> Are we using same SpanManager for retried query or a different one?
Retried queries get their own span manager. This matches the behavior of the debug ui and query profiles.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@180
PS87, Line 180:
> What are we logging here? Do we flush root span in root_->End()?
Good question, will clarify the wording and move it to the end of the function. At this point, the trace is ended and has been sent to the otel sdk code. The otel sdk code may or may not batch it up before sending it to the collector.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@231
PS87, Line 231: void SpanManager::StartChildSpanAdmissionControl() {
> Query won't be retried until it is past admission and actually executing or
On the second query of the retry, client_request_state_->IsRetriedQuery() will be true. The client_request_state_->WasRetried() returns true on the original query when it failed and has a retry query started for it.
http://gerrit.cloudera.org:8080/#/c/22924/87/be/src/observe/span-manager.cc@363
PS87, Line 363: child_span_type_ = span_type;
> It's not clear what <ATTR_RUNNING> = running means since ChildSpanBuilder i
It's a boolean set to true when the query is actually executing (after planning/admission control). Docs will be needed to explain this span attribute.
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.h
File be/src/observe/timed-span.h:
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.h@38
PS89, Line 38:
> We should use string_view as the key since the keys are always constant str
Done.
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.h@58
PS89, Line 58: OtelAttributesMap attributes,
: opentelemetry::trace::SpanKind span_kind =
> We're doing unnecessary copy here. We should not pass these as param but di
I agree it's not the best design. This is another case where root spans have the start time/duration attributes named differently than the child spans. I want to keep TimedSpan agnostic to whether it is a root or a child span. The reason is that future designs may call for more levels where child spans are parents of child spans. Current design only has root spans as parent of child spans. I changed them to string_view. Is that acceptable?
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.cc
File be/src/observe/timed-span.cc:
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.cc@81
PS89, Line 81:
> we should use string_view here.
Done
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.cc@86
PS89, Line 86: ->GetContext().tr
> use string_view here
Done
http://gerrit.cloudera.org:8080/#/c/22924/89/be/src/observe/timed-span.cc@99
PS89, Line 99: } // function End
:
: void TimedSpan::SetAttribute(const nostd::string_view& key,
: const common::AttributeValue& value) noexcept {
: span_->SetAttribute(key, value);
: } // function SetAttribute
:
: void TimedSpan::SetAttributeEmpty(const nostd::string_view& key) noexcept {
: this->SetAttribute(key, "");
: } // function SetAttributeEmpty
:
: void TimedSpan::AddEvent(const string& name, const OtelAttributesMap&
: additional_attributes) noexcept {
: span_->AddEvent(name, additional_attributes);
: } // function AddEvent
:
: trace::Scope TimedSpan::SetActive() {
: return trace::Tracer::WithActiveSpan(span_);
: } // function SetActive
:
: const string& TimedSpan::SpanId() const {
: return span_id_;
: } // function SpanId
:
: const string& TimedSpan::TraceId() const {
: return trace_id_;
: } // function TraceId
:
> If SpanId() and TraceId() are called multiple times for the same span objec
Done, stored them as string but had to return const std::string& since the Substitute() function does not accept opentelemetry::nostd::string_view objects as input.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 90
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sun, 10 Aug 2025 22:13:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#91) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,815 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/91
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 91
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 85:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12289/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 85
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 19:29:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 86:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/observe/otel.cc@243
PS77, Line 243: } else {
: exporter = init_exporter_file();
: }
: VLOG(2) << "OpenTelemetry exporter: " << FLAGS_otel_trace_exporter;
:
> Are we sure that OtlpFileExporterFactory::Create() always returns a non-nul
The OtlpFileExporterFactory::Create() function ends up at https://github.com/open-telemetry/opentelemetry-cpp/blob/5f6d99f0898be265e36f966f8915d8a99c516738/exporters/otlp/src/otlp_file_exporter_factory.cc#L33-L35. The new keyword shouldn't return nullptr unless maybe if the memory allocation fails (I'm not sure if an exception will be thrown or not).
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/runtime/coordinator.cc@155
PS84, Line 155: VLOG_QUERY << "Exec() query_id=" << PrintId(query_id())
> Can we remove this TODO?
Done
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/runtime/coordinator.cc@1497
PS84, Line 1497: vector<NetworkAddressPB> host_addrs;
> Since all the other events are in the past tense, should we call this one "
Yeah, it should be past tense. fixed.
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/scheduling/local-admission-control-client.h
File be/src/scheduling/local-admission-control-client.h:
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/scheduling/local-admission-control-client.h@55
PS84, Line 55: bool was_queued_;
> why do we need to persist this state in the client? I did not see it being
It's used when ending the admission control child span -- https://gerrit.cloudera.org/c/22924/85/be/src/observe/span-manager.cc#286. The other alternative was string parsing of the admission control result.
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/service/client-request-state.cc@1206
PS84, Line 1206: otel_span_manager_->AddChildSpanEvent("QueryUnregistered");
> Should this be "UnregisteredQuery" or "QueryUnregistered" ?
past tense, and I think "QueryUnregistered" sounds better
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/22924/84/be/src/service/impala-server.cc@3610
PS84, Line 3610: if (FLAGS_is_coordinator && otel_trace_enabled()) {
> shutdown condition needs to be same as initialization:
good catch! fixed
http://gerrit.cloudera.org:8080/#/c/22924/84/tests/common/custom_cluster_test_suite.py
File tests/common/custom_cluster_test_suite.py:
http://gerrit.cloudera.org:8080/#/c/22924/84/tests/common/custom_cluster_test_suite.py@692
PS84, Line 692:
: Returns:
> no service in the param, maybe can remove?
done, leftover from previous design
http://gerrit.cloudera.org:8080/#/c/22924/84/tests/common/custom_cluster_test_suite.py@716
PS84, Line 716:
> unreachable break
removed, strange that flake8 did not catch that
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 86
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sat, 09 Aug 2025 04:14:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 86: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 86
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sat, 09 Aug 2025 10:00:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#95) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,839 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/95
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 95
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 94:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22924/94/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/94/be/src/observe/span-manager.cc@388
PS94, Line 388: ATTR_ELAPSED_TIME, std::move(additional_attributes), trace::SpanKind::kInternal, root_);
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/22924/94/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/94/tests/custom_cluster/test_otel_trace.py@26
PS94, Line 26: from tests.util.query_profile_util import parse_db_user, parse_session_id, parse_sql, \
flake8: F401 'tests.util.query_profile_util.parse_num_rows_fetched_from_cache' imported but unused
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 94
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 19:29:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 74:
(4 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22924/74/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/74/tests/custom_cluster/test_otel_trace.py@22
PS74, Line 22: from tests.common.impala_connection import ERROR, RUNNING
flake8: F401 'tests.common.impala_connection.ERROR' imported but unused
http://gerrit.cloudera.org:8080/#/c/22924/74/tests/custom_cluster/test_otel_trace.py@32
PS74, Line 32: class TestOtelTrace(CustomClusterTestSuite):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/22924/74/tests/util/otel_trace.py
File tests/util/otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/74/tests/util/otel_trace.py@59
PS74, Line 59: def __str__(self):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/22924/74/tests/util/otel_trace.py@67
PS74, Line 67: class OtelTrace():
flake8: E302 expected 2 blank lines, found 1
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 74
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Aug 2025 22:10:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 76:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/otel.cc@110
PS76, Line 110: static bool otel_tls_enabled
Can be an inline function.
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/otel.cc@213
PS76, Line 213: static Status
Why not return unique_ptr<SpanExporter> here?
This function never return non-OK status.
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/span-manager.cc@99
PS76, Line 99: (started ? "Started" : "Ended") << " '" << span_name << "' span "
: "trace_id=\"" << span->TraceId() << "\" span_id=\"" << span->SpanId() << "\" "
: "query_id=\"" << query_id << "\""
This can be tidier with Substitute(). Same comment for others below.
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/span-manager.cc@290
PS76, Line 290: ""
Why is it empty string and not "Admitted immediately"?
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/timed-span.cc
File be/src/observe/timed-span.cc:
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/timed-span.cc@39
PS76, Line 39: long long
int64_t is preferable than long long.
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/runtime/coordinator.cc@612
PS76, Line 612: AllBackendsStarted
Why no space in between words?
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/runtime/coordinator.cc@1499
PS76, Line 1499: Request admission control
Should this be "Released admission control resources"?
Please match with the string printed in query profile whenever possible to make association easy.
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/service/client-request-state.cc@789
PS76, Line 789: UpdateCatalogFinished
Why no spaces in between words? Same comment for few similar cases below.
http://gerrit.cloudera.org:8080/#/c/22924/76/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/76/tests/custom_cluster/test_otel_trace.py@220
PS76, Line 220: 5, 1, 2
nit: mention the argument name for clarity.
max_attempts=5, sleep_time_s=1, backoff=2)
http://gerrit.cloudera.org:8080/#/c/22924/76/tests/util/otel_trace.py
File tests/util/otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/76/tests/util/otel_trace.py@59
PS76, Line 59: def __str__(self):
> flake8: E302 expected 2 blank lines, found 1
Please fix this.
http://gerrit.cloudera.org:8080/#/c/22924/76/tests/util/otel_trace.py@67
PS76, Line 67: class OtelTrace():
> flake8: E302 expected 2 blank lines, found 1
Please fix this.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 76
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Thu, 07 Aug 2025 23:42:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#79) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M CMakeLists.txt
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,814 insertions(+), 12 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/79
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 79
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 79:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20089/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 79
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 03:57:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#76) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M CMakeLists.txt
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,798 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/76
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 76
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 2:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19279/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 2
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 May 2025 18:25:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 6:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/19399/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 6
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Jun 2025 18:54:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19335/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 3
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 May 2025 19:24:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13235: Update the build toolchain to include the latest OpenTelemetry C++ sdk.
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19412/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 8
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Jun 2025 23:51:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5 of 5] - Implement OpenTelemetry Traces for Query Tracking
......................................................................
Patch Set 70:
(7 comments)
Thanks for the review Riza!
http://gerrit.cloudera.org:8080/#/c/22924/68//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/22924/68//COMMIT_MSG@39
PS68, Line 39:
: Once the query has closed, the root span is closed.
> What happen if query terminate abnormally or cancelled?
In that case, the current child and root spans are ended. No further child spans are started.
For example, if the query has a sql syntax error, the OTel trace will only have the init, submitted, and planning child spans on it.
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.h
File be/src/service/client-request-state.h:
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.h@551
PS68, Line 551: return static_cast<bool>(otel_span_manager_);
> Isn't it better this way?
That definitely is the pattern. I didn't find any other instances of casting a shared_ptr to a bool. Fixed.
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.h@551
PS68, Line 551: otel_trace
> nit: is_otel_enabled sounds better.
I've never been a fan of the "is_" prefix. To me, it does not add clarity. I changed the name of this method to otel_trace_enabled() though as that is much clearer.
If support for other types of otel data collection (e.g. metrics/logs) is added in the future, it will be desirable to have separate flags for each as some users may only want 1 or 2 types of otel data.
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.cc@128
PS68, Line 128: && !boost::algorithm::istarts_with(sql_stmt().c_str(), ":")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "copy ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "describe ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "explain ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "load ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "set ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "show ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "use ")
: && !boost::algorithm::istarts_with(sql_stmt().c_str(), "values ")
> Create a function for this? Is there any better way to detect the kind of q
sql_stmt is always trimmed. I both tested it and confirmed in the code.
At this point in the query lifecycle, there is no better way since query type is not determined until planning..
Moved this code to a separate function and switched it to be allow list style instead of deny list style.
http://gerrit.cloudera.org:8080/#/c/22924/68/be/src/service/client-request-state.cc@1288
PS68, Line 1288: otel_span_manager_->EndChildSpanQueryExecution();
: otel_span_manager()->StartChildSpanClose();
> Consistently use otel_span_manager_ everywhere.
Done
http://gerrit.cloudera.org:8080/#/c/22924/68/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/68/tests/custom_cluster/test_otel_trace.py@114
PS68, Line 114: query_id, profile = self.query_id_from_ui(section="completed_queries",
: match_query=query)
> Is it not possible to collect profile through self.client.get_runtime_profi
It's not possible (that I know of) to get the runtime profile when using self.execute_query_expect_failure because it returns a HiveServer2Error object instead of returning the handle. I added a comment explaining this.
I could use self.execute_query_async to get the handle, but that requires more code and also catching exceptions. I'm not opposed to that way, I merely view this way as cleaner.
http://gerrit.cloudera.org:8080/#/c/22924/68/tests/custom_cluster/test_otel_trace.py@249
PS68, Line 249: self.query_profile_from_ui(result.query_id)
> Is it not possible to collect profile through self.client.get_runtime_profi
Unfortunately no. When a query is retried, a second query is launched. The query profile returned to the client is mainly the profile of that second query but has the id of the original query (plus a few other attributes like Retry Status/Cause. The query profile of the original query in the Impala UI shows the state as exception. The profile returned to the client does not show that state.
One otel trace will be created for each query, and the attributes on each trace match the attributes in the query profiles in the Impala UI, not necessarily the query profile returned to the client.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 70
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Aug 2025 22:21:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#75) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
27 files changed, 2,797 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/75
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 75
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#74) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
27 files changed, 2,796 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/74
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 74
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 77:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/otel.cc@110
PS76, Line 110: static inline bool otel_tls_
> Can be an inline function.
Done
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/otel.cc@213
PS76, Line 213: static unique
> Why not return unique_ptr<SpanExporter> here?
Done. It was a leftover from a previous design.
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/span-manager.cc@99
PS76, Line 99: an != nullptr)) {
: VLOG(2) << strings::Substitute("$0 '$1' span trace_id=\"$2\" span_id=\"$3\" "
: "query_id=\"$4\"", (started ? "St
> This can be tidier with Substitute(). Same comment for others below.
Done
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/span-manager.cc@290
PS76, Line 290: li
> Why is it empty string and not "Admitted immediately"?
The design originally called for including only the queued reason. After reading this comment and giving the design more thought, I want to match the query profile. Thus, this attribute is now "AdmissionResult". and contains the admission result string unchanged.
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/timed-span.cc
File be/src/observe/timed-span.cc:
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/observe/timed-span.cc@39
PS76, Line 39: int64_t g
> int64_t is preferable than long long.
Done
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/runtime/coordinator.cc@612
PS76, Line 612: AllBackendsStarted
> Why no space in between words?
Purely a judgment call. I prefer keep spaces out of it because it's easier for scripts to parse the text that way.
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/runtime/coordinator.cc@1499
PS76, Line 1499: ReleaseAdmissionControlRe
> Should this be "Released admission control resources"?
Done
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/22924/76/be/src/service/client-request-state.cc@789
PS76, Line 789: UpdateCatalogFinished
> Why no spaces in between words? Same comment for few similar cases below.
see https://gerrit.cloudera.org/c/22924/76/be/src/runtime/coordinator.cc#612
http://gerrit.cloudera.org:8080/#/c/22924/76/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/76/tests/custom_cluster/test_otel_trace.py@220
PS76, Line 220:
> nit: mention the argument name for clarity.
Done
http://gerrit.cloudera.org:8080/#/c/22924/76/tests/util/otel_trace.py
File tests/util/otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/76/tests/util/otel_trace.py@59
PS76, Line 59: def __str__(self):
> Please fix this.
Done
http://gerrit.cloudera.org:8080/#/c/22924/76/tests/util/otel_trace.py@67
PS76, Line 67:
> Please fix this.
Done
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 77
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 00:47:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 80:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/service/client-request-state.cc@1266
PS79, Line 1266: RowsAvailable")
> nit: RowsAvailable
Done
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 80
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 04:20:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 80: Code-Review+1
Please go ahead rebase and DRY_RUN this patch while waiting for another reviewer.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 80
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 04:22:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 80:
(20 comments)
Mainly nits...
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/observe/otel-flags-trace.cc
File be/src/observe/otel-flags-trace.cc:
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/observe/otel-flags-trace.cc@75
PS77, Line 75: "http://"
nit. define const for string values?
static const string HTTP_PREFIX = "http://";
static const string HTTPS_PREFIX = "https://";
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/observe/otel-flags-trace.cc@162
PS77, Line 162: value == "1.2" || value == "1.3"
maybe we can put these in a global set?
static const set<string> VALID_TLS_VERSIONS = {"1.2", "1.3"};
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/observe/otel-flags-trace.cc@165
PS77, Line 165: '1.2', '1.3'
If we are using a set for versions, we also need to change this too.
http://gerrit.cloudera.org:8080/#/c/22924/80/be/src/observe/otel-flags-trace.cc
File be/src/observe/otel-flags-trace.cc:
http://gerrit.cloudera.org:8080/#/c/22924/80/be/src/observe/otel-flags-trace.cc@85
PS80, Line 85: delimeter
nit. delimiter
http://gerrit.cloudera.org:8080/#/c/22924/80/be/src/observe/otel-flags-trace.cc@159
PS80, Line 159: ’
nit. '
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/observe/otel.cc@152
PS77, Line 152: zlib
nit. maybe we can define the constants for these strings? Same for the "http_exporter" and "batch_span_processor".
static const string COMPRESSION_ZLIB = "zlib";
static const string HTTP_EXPORTER_NAME = "http_exporter";
static const string BATCH_PROCESSOR_NAME = "batch_span_processor";
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/observe/otel.cc@243
PS77, Line 243: if(FLAGS_otel_trace_exporter == OTEL_EXPORTER_OTLP_HTTP) {
: RETURN_IF_ERROR(init_exporter_http(exporter));
: } else {
: exporter = init_exporter_file();
: }
Seems better to check whether exporter is nullptr after init, and do some action if nullptr
http://gerrit.cloudera.org:8080/#/c/22924/80/be/src/observe/otel.cc
File be/src/observe/otel.cc:
http://gerrit.cloudera.org:8080/#/c/22924/80/be/src/observe/otel.cc@209
PS80, Line 209: OtlpHttpExporter
should be OtlpFileExporter?
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/observe/span-manager.h
File be/src/observe/span-manager.h:
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/observe/span-manager.h@80
PS79, Line 80: behvaior
nit. behavior
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/observe/span-manager.h@104
PS79, Line 104: deactive
nit. deactivate
http://gerrit.cloudera.org:8080/#/c/22924/79/be/src/observe/span-manager.h@150
PS79, Line 150: it's
nit. its
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/service/impala-server.cc@1332
PS77, Line 1332:
nit. unnecessary change
http://gerrit.cloudera.org:8080/#/c/22924/77/be/src/service/impala-server.cc@1511
PS77, Line 1511:
nit. unnecessary change
http://gerrit.cloudera.org:8080/#/c/22924/80/tests/common/custom_cluster_test_suite.py
File tests/common/custom_cluster_test_suite.py:
http://gerrit.cloudera.org:8080/#/c/22924/80/tests/common/custom_cluster_test_suite.py@682
PS80, Line 682: valie
nit. valid
http://gerrit.cloudera.org:8080/#/c/22924/80/tests/common/custom_cluster_test_suite.py@692
PS80, Line 692: who's
nit. whose
http://gerrit.cloudera.org:8080/#/c/22924/80/tests/common/custom_cluster_test_suite.py@722
PS80, Line 722: funtion
nit. function
http://gerrit.cloudera.org:8080/#/c/22924/80/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/80/tests/custom_cluster/test_otel_trace.py@415
PS80, Line 415: line line
nit. line
http://gerrit.cloudera.org:8080/#/c/22924/79/tests/util/otel_trace.py
File tests/util/otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/79/tests/util/otel_trace.py@35
PS79, Line 35: Additionaly
nit. Additionally
http://gerrit.cloudera.org:8080/#/c/22924/79/tests/util/otel_trace.py@117
PS79, Line 117: extraced
nit. extracted
http://gerrit.cloudera.org:8080/#/c/22924/79/tests/util/query_profile_util.py
File tests/util/query_profile_util.py:
http://gerrit.cloudera.org:8080/#/c/22924/79/tests/util/query_profile_util.py@82
PS79, Line 82: was
The sentence not finished, not retried?
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 80
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Aug 2025 04:49:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#87) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,810 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/87
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 87
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 87:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12294/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 87
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sat, 09 Aug 2025 14:43:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#92) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,814 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/92
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 92
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 90:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20113/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 90
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Sun, 10 Aug 2025 22:39:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#89) to the change originally created by gaurav singh. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Adds representation of Impala select queries using OpenTelemetry
traces.
Each Impala query is represented as its own individual OpenTelemetry
trace. The one exception is retried queries which will have an
individual trace for each attempt. These traces consist of a root span
and several child spans. Each child span has the root as its parent.
No child span has another child span as its parent. Each child span
represents one high-level query lifecycle stage. Each child span also
has span attributes that further describe the state of the query.
Child spans:
1. Init
2. Submitted
3. Planning
4. Admission Control
5. Query Execution
6. Close
Each child span contains a mix of universal attributes (available on
all spans) and query phase specific attributes. For example, the
"ErrorMsg" attribute, present on all child spans, is the error
message (if any) at the end of that particular query phase. One
example of a child span specific attribute is "QueryType" on the
Planning span. Since query type is first determined during query
planning, the "QueryType" attribute is present on the Planning span
and has a value of "QUERY" (since only selects are supported).
Since queries can run for lengthy periods of time, the Init span
communicates the beginning of a query along with global query
attributes. For example, span attributes include query id, session
id, sql, user, etc.
Once the query has closed, the root span is closed.
Testing accomplished with new custom cluster tests.
Generated-by: Github Copilot (GPT-4.1, Claude Sonnet 3.7)
Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
---
M be/CMakeLists.txt
A be/src/observe/CMakeLists.txt
A be/src/observe/otel-flags-trace.cc
A be/src/observe/otel-flags.cc
A be/src/observe/otel-instrument.h
A be/src/observe/otel.cc
A be/src/observe/otel.h
A be/src/observe/span-manager.cc
A be/src/observe/span-manager.h
A be/src/observe/timed-span.cc
A be/src/observe/timed-span.h
M be/src/runtime/coordinator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-driver.cc
M be/src/scheduling/admission-control-client.h
M be/src/scheduling/local-admission-control-client.cc
M be/src/scheduling/local-admission-control-client.h
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-server.cc
M bin/impala-config.sh
M tests/common/custom_cluster_test_suite.py
M tests/common/file_utils.py
A tests/custom_cluster/test_otel_trace.py
A tests/util/otel_trace.py
A tests/util/query_profile_util.py
28 files changed, 2,817 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/22924/89
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 89
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 96:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22924/96/be/src/observe/span-manager.cc
File be/src/observe/span-manager.cc:
http://gerrit.cloudera.org:8080/#/c/22924/96/be/src/observe/span-manager.cc@388
PS96, Line 388: ATTR_ELAPSED_TIME, std::move(additional_attributes), trace::SpanKind::kInternal, root_);
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/22924/96/tests/custom_cluster/test_otel_trace.py
File tests/custom_cluster/test_otel_trace.py:
http://gerrit.cloudera.org:8080/#/c/22924/96/tests/custom_cluster/test_otel_trace.py@26
PS96, Line 26: from tests.util.query_profile_util import parse_db_user, parse_session_id, parse_sql, \
flake8: F401 'tests.util.query_profile_util.parse_num_rows_fetched_from_cache' imported but unused
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 96
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 19:35:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 98: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 98
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 22:58:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22924 )
Change subject: IMPALA-13237: [Patch 5] - Implement OpenTelemetry Traces for Select Queries Tracking
......................................................................
Patch Set 94:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20122/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40b5cd33274df13f3005bf7a704299ebfff8a5b
Gerrit-Change-Number: 22924
Gerrit-PatchSet: 94
Gerrit-Owner: gaurav singh <gs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Reviewer: gaurav singh <gs...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Aug 2025 19:53:41 +0000
Gerrit-HasComments: No