You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2026/05/02 00:58:32 UTC

[Impala-ASF-CR] IMPALA-14843: [WIP] Show cancelled nodes in ExecSummary

Hello Jason Fehr, Surya Hebbar, Daniel Vanko, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24118

to look at the new patch set (#10).

Change subject: IMPALA-14843: [WIP] Show cancelled nodes in ExecSummary
......................................................................

IMPALA-14843: [WIP] Show cancelled nodes in ExecSummary

In backend execution, a plan node instance could be closed before it
produces all its output rows, e.g., when a UnionNode reaches its output
limit, the children operands are all closed regardless what their states
(prepared, opened, executing, etc.) are. In ExecSummary shown in the
query profile or WebUI, such nodes usually have 0 as the output
cardinality without any indication, which is confusing. This patch adds
a "cancelled" marker in the Detail column to represent such nodes, i.e.,
nodes that haven't reached the "Last Batch Returned" state.

Implementation:
 - Adds a last_batch_returned_ flag in ExecNode to indicate whether the
   "Last Batch Returned" event has been added for this node.
 - Extends ExecSummaryDataPB and TExecStats to add a last_batch_returned
   field to reflect such case. Note that ExecSummaryDataPB is used in
   executors reporting execution status to the coordinator. TExecStats
   is used in coordinator to track execution stats for plan nodes.
 - Executor mirrors the flag to ExecSummaryDataPB for each node when
   generating status report for coordinator. To lookup the ExecNode
   instance based on the plan node id, a map is added in
   FragmentInstanceState when the ExecNode tree is built.
 - Coordinator appends the "cancelled" marker when returning
   TExecSummary.

Example:
For query
 "with l as
    (select * from tpch.lineitem UNION ALL select * from tpch.lineitem)
  select STRAIGHT_JOIN count(*)
  from (select * from tpch.lineitem a LIMIT 1) a
  join (select * from l LIMIT 125000) b
  on a.l_orderkey = -b.l_orderkey"

the ExecSummary is
Operator                 #Hosts  #Inst   Avg Time   Max Time    #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail
---------------------------------------------------------------------------------------------------------------------------------------
F01:ROOT                      1      1    8.647us    8.647us                               0              0
05:AGGREGATE                  1      1    0.000ns    0.000ns        1           1   24.00 KB       16.00 KB  FINALIZE
04:HASH JOIN                  1      1  251.664us  251.664us        0           1    9.06 MB        4.75 MB  INNER JOIN, BROADCAST
|--08:EXCHANGE                1      1  427.557us  427.557us  125.00K     125.00K  232.00 KB      337.52 KB  UNPARTITIONED
|  F05:EXCHANGE SENDER        1      1    3.007ms    3.007ms                        54.05 KB       48.00 KB
|  07:EXCHANGE                1      1  499.833us  499.833us  125.00K     125.00K    5.12 MB      361.52 KB  UNPARTITIONED
|  F04:EXCHANGE SENDER        3      3    2.681ms    2.833ms                        54.05 KB       48.00 KB
|  01:UNION                   3      3   43.817us   48.670us  375.00K     125.00K          0              0
|  |--03:SCAN HDFS            3      3   40.710us   49.726us        0       6.00M          0      264.00 MB  tpch.lineitem (cancelled)
|  02:SCAN HDFS               3      3    5.813ms    7.886ms  377.86K       6.00M   48.23 MB      264.00 MB  tpch.lineitem (cancelled)
06:EXCHANGE                   1      1   10.288us   10.288us        1           1   16.00 KB       16.00 KB  UNPARTITIONED
F00:EXCHANGE SENDER           3      3   32.298us   35.572us                         31.00 B       48.00 KB
00:SCAN HDFS                  3      3   76.692ms   85.439ms        3           1   48.07 MB      264.00 MB  tpch.lineitem a

Note that "02:SCAN HDFS" and "03:SCAN HDFS" are cancelled due to
"01:UNION" reaches its limit.

Test:
 - Added e2e tests in query_test/test_observability.py
 - TODO: cover more node types like UnnestNode, SystemTableScanNode,
   IcebergMetadataScanNode, KuduScanNodeMt that seem to don't have
   "Last batch returned" event.

Assisted-by: Claude Sonnet 4.5
Change-Id: Ibd1bf51112e7c8c5609bb9e9525d960efc524215
---
M be/src/exec/exec-node-util.h
M be/src/exec/exec-node.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/fragment-instance-state.cc
M be/src/runtime/fragment-instance-state.h
M be/src/service/impala-http-handler.cc
M be/src/util/summary-util.cc
M be/src/util/summary-util.h
M common/protobuf/control_service.proto
M common/thrift/ExecStats.thrift
M tests/query_test/test_observability.py
13 files changed, 132 insertions(+), 8 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/24118/10
-- 
To view, visit http://gerrit.cloudera.org:8080/24118
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibd1bf51112e7c8c5609bb9e9525d960efc524215
Gerrit-Change-Number: 24118
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Daniel Vanko <dv...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Surya Hebbar <sh...@cloudera.com>