You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2026/05/02 00:58:32 UTC
[Impala-ASF-CR] IMPALA-14843: [WIP] Show cancelled nodes in ExecSummary
Hello Jason Fehr, Surya Hebbar, Daniel Vanko, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24118
to look at the new patch set (#10).
Change subject: IMPALA-14843: [WIP] Show cancelled nodes in ExecSummary
......................................................................
IMPALA-14843: [WIP] Show cancelled nodes in ExecSummary
In backend execution, a plan node instance could be closed before it
produces all its output rows, e.g., when a UnionNode reaches its output
limit, the children operands are all closed regardless what their states
(prepared, opened, executing, etc.) are. In ExecSummary shown in the
query profile or WebUI, such nodes usually have 0 as the output
cardinality without any indication, which is confusing. This patch adds
a "cancelled" marker in the Detail column to represent such nodes, i.e.,
nodes that haven't reached the "Last Batch Returned" state.
Implementation:
- Adds a last_batch_returned_ flag in ExecNode to indicate whether the
"Last Batch Returned" event has been added for this node.
- Extends ExecSummaryDataPB and TExecStats to add a last_batch_returned
field to reflect such case. Note that ExecSummaryDataPB is used in
executors reporting execution status to the coordinator. TExecStats
is used in coordinator to track execution stats for plan nodes.
- Executor mirrors the flag to ExecSummaryDataPB for each node when
generating status report for coordinator. To lookup the ExecNode
instance based on the plan node id, a map is added in
FragmentInstanceState when the ExecNode tree is built.
- Coordinator appends the "cancelled" marker when returning
TExecSummary.
Example:
For query
"with l as
(select * from tpch.lineitem UNION ALL select * from tpch.lineitem)
select STRAIGHT_JOIN count(*)
from (select * from tpch.lineitem a LIMIT 1) a
join (select * from l LIMIT 125000) b
on a.l_orderkey = -b.l_orderkey"
the ExecSummary is
Operator #Hosts #Inst Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
---------------------------------------------------------------------------------------------------------------------------------------
F01:ROOT 1 1 8.647us 8.647us 0 0
05:AGGREGATE 1 1 0.000ns 0.000ns 1 1 24.00 KB 16.00 KB FINALIZE
04:HASH JOIN 1 1 251.664us 251.664us 0 1 9.06 MB 4.75 MB INNER JOIN, BROADCAST
|--08:EXCHANGE 1 1 427.557us 427.557us 125.00K 125.00K 232.00 KB 337.52 KB UNPARTITIONED
| F05:EXCHANGE SENDER 1 1 3.007ms 3.007ms 54.05 KB 48.00 KB
| 07:EXCHANGE 1 1 499.833us 499.833us 125.00K 125.00K 5.12 MB 361.52 KB UNPARTITIONED
| F04:EXCHANGE SENDER 3 3 2.681ms 2.833ms 54.05 KB 48.00 KB
| 01:UNION 3 3 43.817us 48.670us 375.00K 125.00K 0 0
| |--03:SCAN HDFS 3 3 40.710us 49.726us 0 6.00M 0 264.00 MB tpch.lineitem (cancelled)
| 02:SCAN HDFS 3 3 5.813ms 7.886ms 377.86K 6.00M 48.23 MB 264.00 MB tpch.lineitem (cancelled)
06:EXCHANGE 1 1 10.288us 10.288us 1 1 16.00 KB 16.00 KB UNPARTITIONED
F00:EXCHANGE SENDER 3 3 32.298us 35.572us 31.00 B 48.00 KB
00:SCAN HDFS 3 3 76.692ms 85.439ms 3 1 48.07 MB 264.00 MB tpch.lineitem a
Note that "02:SCAN HDFS" and "03:SCAN HDFS" are cancelled due to
"01:UNION" reaches its limit.
Test:
- Added e2e tests in query_test/test_observability.py
- TODO: cover more node types like UnnestNode, SystemTableScanNode,
IcebergMetadataScanNode, KuduScanNodeMt that seem to don't have
"Last batch returned" event.
Assisted-by: Claude Sonnet 4.5
Change-Id: Ibd1bf51112e7c8c5609bb9e9525d960efc524215
---
M be/src/exec/exec-node-util.h
M be/src/exec/exec-node.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/fragment-instance-state.cc
M be/src/runtime/fragment-instance-state.h
M be/src/service/impala-http-handler.cc
M be/src/util/summary-util.cc
M be/src/util/summary-util.h
M common/protobuf/control_service.proto
M common/thrift/ExecStats.thrift
M tests/query_test/test_observability.py
13 files changed, 132 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/24118/10
--
To view, visit http://gerrit.cloudera.org:8080/24118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibd1bf51112e7c8c5609bb9e9525d960efc524215
Gerrit-Change-Number: 24118
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>