You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org> on 2026/08/14 09:41:56 UTC
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Aleksandr Efimov has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24693
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
Record the canonicalization strategy and HBO hash together with the
matched cardinality. Show them in the plan profile and keep the original
estimate for debugging.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance. Keep the empty-scan matching introduced by
IMPALA-15127, including constant-only UNION plans.
Testing:
- HistoricalStatsTest
- Focused HBO runtime validation
- python3 bin/jenkins/critique-gerrit-review.py --dryrun
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: gpt-5 (OpenAI Codex)
---
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-collection-scan.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-disjunct-selectivity.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-distinct-agg.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-grouping-set.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-iceberg-scan.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-agg.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-nonpartitioned-no-stats.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-nonpartitioned-stats.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-no-stats.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-union-collection-scan.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-union-consts.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-union-same-tables.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
19 files changed, 367 insertions(+), 231 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/24693/1
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 1
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 12: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 12
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Sep 2026 00:27:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Record the match on the PlanNode and add a details line next to the
cardinality naming the canonicalization strategy that matched, the
opaque hash key, the estimate the planner had computed, and the ratio
between the two, so a tool reading the line does not have to parse
"7.30K" back into a number. The line shows at EXTENDED and above. That
keeps the default EXPLAIN text from growing a line per plan node, and
still leaves the details in the profile of an ordinary query:
statements other than EXPLAIN render their plan at EXTENDED, and the
profile is what remains once the query is gone.
At EXTENDED, a scan whose estimate came from a recorded run reads:
00:SCAN HDFS [functional.alltypes, RANDOM]
partition predicates: `year` = CAST(2009 AS INT)
HDFS partitions=12/24 files=12 size=238.68KB
predicates: int_col = CAST(1 AS INT), string_col = '1'
stored statistics:
table: rows=7.30K size=478.45KB
partitions: 12/12 rows=3.65K
columns: all
extrapolated-rows=disabled max-scan-range-rows=622
mem-estimate=80.00MB mem-reservation=32.00KB thread-reservation=1
tuple-ids=0 row-size=17B cardinality=365 (from HBO)
HBO match: strategy=EXPR_REWRITE, hash=56704b19611ac2584658f691edebdf22, original estimate=115, ratio=3.17
in pipelines: 00(GETNEXT)
The planner had 115 rows for that scan on its own, and the run it
matched had 365.
The line is text rather than fields on TPlanNode next to the hbo_hash_keys
that already travel there. Those keys are written when store_hbo_stats is
on, to record a run; this says what a lookup found while planning, and it
is read where the estimate itself is read - in the plan, in the profile,
and in whatever a person pastes into a ticket. A structured form of it is
worth having the day something other than a reader wants it.
Not every match deserves the same trust, and that belongs next to the
number rather than an explain level away. IGNORE_PARTITION_CONSTANTS
drops the constants from partition equality predicates and assumes all
partitions look alike, so the run it matched may have been recorded for
a different partition than the one being planned. Such a match now reads
"(from HBO, key ignores partition constants)". An EXPR_REWRITE match
keeps its constants and describes the query being planned, so it stays
"(from HBO)" and the default EXPLAIN text is unchanged there.
The caveat says what the key leaves out, not what the node does, because
the key covers the subtree: an aggregate over a scan on partition
predicates carries it while having no partition predicates of its own,
which is what the goldens here show.
The ratio goes through a new PrintUtils.printTwoDecimalsRatio, which
formats under Locale.ROOT. It is there for a tool to read rather than
for the eye, and a coordinator whose JVM default is, say, de_DE would
write 0,05 instead. The cardinalities beside it use the default locale
like every other estimate in a plan, so this is about the one field
that claims to be machine-readable, not about the line as a whole.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- PrintUtilsTest, HboKeyStringTest, HistoricalStatsTest: 28 tests, 0
failures. Formatting the ratio under the default locale instead fails
testPrintTwoDecimalsRatio alone, with "expected:<0[.]05> but
was:<0[,]05>"
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24693
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/common/PrintUtils.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M fe/src/test/java/org/apache/impala/util/PrintUtilsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M tests/query_test/test_hbo.py
9 files changed, 239 insertions(+), 30 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Quanlong Huang: Looks good to me, but someone else must approve
Michael Smith: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 13
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 5:
(2 comments)
The profile still carries this as text. IMPALA-15236 mentions plan-graph.py not being able to explain why history was reused, and the accuracy rate you asked for is the same direction. Do you want a machine-readable form tracked here, or as its own change? The Jira says it does not require stabilising the V2 profile format, so I read it as separate work, but it is your call.
Tested: HistoricalStatsTest, 8 tests, and the frontend builds. I could not run test_matching_provenance - no cluster here - so the profile behaviour is verified by reading and compiling rather than by running.
http://gerrit.cloudera.org:8080/#/c/24693/5/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/5/fe/src/main/java/org/apache/impala/planner/PlanNode.java@491
PS5, Line 491: expBuilder.append(detailPrefix)
My answer here on PS2 was wrong and I should not have marked it done. The option gated the details line, but the strategy went into the annotation itself, so "(from HBO)" became "(from HBO EXPR_REWRITE)" on 218 golden lines.
PS5 keeps the annotation stable and moves the strategy into the details line - the place you called it redundant on PS2. It is not redundant now, since the annotation no longer names it. The one default-text change left is a match that dropped partition constants: it may describe a different partition than the one being planned, so it reads "(from HBO, partition constants ignored)". In the goldens that is 204 lines back to their old text, 14 changed, and thirteen golden files out of the patch.
If you would rather have no change there at all, it can go behind the option too.
http://gerrit.cloudera.org:8080/#/c/24693/5/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24693/5/tests/query_test/test_hbo.py@92
PS5, Line 92: # behind once the query is gone, and it names the strategy that matched.
Done in PS5. The profile is rendered by its own call that forces the details on, so it carries them with no query option, while EXPLAIN still needs ENABLE_EXPLAIN_HBO. The second render only happens when the two would differ - the option off and some node carrying an HBO match - so a query without HBO pays nothing for it.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sun, 23 Aug 2026 11:47:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24693
to look at the new patch set (#10).
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Record the match on the PlanNode and add a details line next to the
cardinality naming the canonicalization strategy that matched, the
opaque hash key, the estimate the planner had computed, and the ratio
between the two, so a tool reading the line does not have to parse
"7.30K" back into a number. The line shows at EXTENDED and above. That
keeps the default EXPLAIN text from growing a line per plan node, and
still leaves the details in the profile of an ordinary query:
statements other than EXPLAIN render their plan at EXTENDED, and the
profile is what remains once the query is gone.
Not every match deserves the same trust, and that belongs next to the
number rather than an explain level away. IGNORE_PARTITION_CONSTANTS
drops the constants from partition equality predicates and assumes all
partitions look alike, so the run it matched may have been recorded for
a different partition than the one being planned. Such a match now reads
"(from HBO, partition constants ignored)". An EXPR_REWRITE match keeps
its constants and describes the query being planned, so it stays
"(from HBO)" and the default EXPLAIN text is unchanged there.
The ratio goes through a new PrintUtils.printTwoDecimalsRatio, which
formats under Locale.ROOT. It is there for a tool to read rather than
for the eye, and a coordinator whose JVM default is, say, de_DE would
write 0,05 instead. The cardinalities beside it use the default locale
like every other estimate in a plan, so this is about the one field
that claims to be machine-readable, not about the line as a whole.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- PrintUtilsTest, HboKeyStringTest, HistoricalStatsTest: 28 tests, 0
failures. Formatting the ratio under the default locale instead fails
testPrintTwoDecimalsRatio alone, with "expected:<0[.]05> but
was:<0[,]05>"
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/common/PrintUtils.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M fe/src/test/java/org/apache/impala/util/PrintUtilsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M tests/query_test/test_hbo.py
9 files changed, 233 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/24693/10
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 10
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24693
to look at the new patch set (#12).
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Record the match on the PlanNode and add a details line next to the
cardinality naming the canonicalization strategy that matched, the
opaque hash key, the estimate the planner had computed, and the ratio
between the two, so a tool reading the line does not have to parse
"7.30K" back into a number. The line shows at EXTENDED and above. That
keeps the default EXPLAIN text from growing a line per plan node, and
still leaves the details in the profile of an ordinary query:
statements other than EXPLAIN render their plan at EXTENDED, and the
profile is what remains once the query is gone.
At EXTENDED, a scan whose estimate came from a recorded run reads:
00:SCAN HDFS [functional.alltypes, RANDOM]
partition predicates: `year` = CAST(2009 AS INT)
HDFS partitions=12/24 files=12 size=238.68KB
predicates: int_col = CAST(1 AS INT), string_col = '1'
stored statistics:
table: rows=7.30K size=478.45KB
partitions: 12/12 rows=3.65K
columns: all
extrapolated-rows=disabled max-scan-range-rows=622
mem-estimate=80.00MB mem-reservation=32.00KB thread-reservation=1
tuple-ids=0 row-size=17B cardinality=365 (from HBO)
HBO match: strategy=EXPR_REWRITE, hash=56704b19611ac2584658f691edebdf22, original estimate=115, ratio=3.17
in pipelines: 00(GETNEXT)
The planner had 115 rows for that scan on its own, and the run it
matched had 365.
The line is text rather than fields on TPlanNode next to the hbo_hash_keys
that already travel there. Those keys are written when store_hbo_stats is
on, to record a run; this says what a lookup found while planning, and it
is read where the estimate itself is read - in the plan, in the profile,
and in whatever a person pastes into a ticket. A structured form of it is
worth having the day something other than a reader wants it.
Not every match deserves the same trust, and that belongs next to the
number rather than an explain level away. IGNORE_PARTITION_CONSTANTS
drops the constants from partition equality predicates and assumes all
partitions look alike, so the run it matched may have been recorded for
a different partition than the one being planned. Such a match now reads
"(from HBO, key ignores partition constants)". An EXPR_REWRITE match
keeps its constants and describes the query being planned, so it stays
"(from HBO)" and the default EXPLAIN text is unchanged there.
The caveat says what the key leaves out, not what the node does, because
the key covers the subtree: an aggregate over a scan on partition
predicates carries it while having no partition predicates of its own,
which is what the goldens here show.
The ratio goes through a new PrintUtils.printTwoDecimalsRatio, which
formats under Locale.ROOT. It is there for a tool to read rather than
for the eye, and a coordinator whose JVM default is, say, de_DE would
write 0,05 instead. The cardinalities beside it use the default locale
like every other estimate in a plan, so this is about the one field
that claims to be machine-readable, not about the line as a whole.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- PrintUtilsTest, HboKeyStringTest, HistoricalStatsTest: 28 tests, 0
failures. Formatting the ratio under the default locale instead fails
testPrintTwoDecimalsRatio alone, with "expected:<0[.]05> but
was:<0[,]05>"
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/common/PrintUtils.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M fe/src/test/java/org/apache/impala/util/PrintUtilsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M tests/query_test/test_hbo.py
9 files changed, 239 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/24693/12
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 12
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24693/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24693/11//COMMIT_MSG@14
PS11, Line 14: Record the match on the PlanNode and add a details line next to the
> nit: it'd be helpful to paste an example of a PlanNode in the commit messag
Done in PS12 - the scan node from a cluster run, with the details line under the cardinality: the planner had 115 rows there and the run it matched had 365.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 11
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2026 20:01:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13939/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 8
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Aug 2026 09:34:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23721/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 4
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sun, 23 Aug 2026 10:56:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 10: Code-Review+1
Thanks for addressing the comments! LGTM. Let's wait some time to see if other reviewers have more comments.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 10
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2026 00:51:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 12: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 12
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Sep 2026 20:59:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24693
to look at the new patch set (#9).
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Record the match on the PlanNode and add a details line next to the
cardinality naming the canonicalization strategy that matched, the
opaque hash key, the estimate the planner had computed, and the ratio
between the two, so a tool reading the line does not have to parse
"7.30K" back into a number. The line shows at EXTENDED and above. That
keeps the default EXPLAIN text from growing a line per plan node, and
still leaves the details in the profile of an ordinary query:
statements other than EXPLAIN render their plan at EXTENDED, and the
profile is what remains once the query is gone.
Not every match deserves the same trust, and that belongs next to the
number rather than an explain level away. IGNORE_PARTITION_CONSTANTS
drops the constants from partition equality predicates and assumes all
partitions look alike, so the run it matched may have been recorded for
a different partition than the one being planned. Such a match now reads
"(from HBO, partition constants ignored)". An EXPR_REWRITE match keeps
its constants and describes the query being planned, so it stays
"(from HBO)" and the default EXPLAIN text is unchanged there.
The ratio is formatted under Locale.ROOT. It is there for a tool to
read rather than for the eye, and a coordinator whose JVM default is,
say, de_DE would write 0,05 instead. The cardinalities beside it go
through PrintUtils, which uses the default locale like every other
estimate in a plan, so this is about the one field that claims to be
machine-readable, not about the line as a whole.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- HistoricalStatsTest
- HboKeyStringTest: 15 tests, 0 failures, including the new
testHboRatioIsLocaleIndependent; putting the DecimalFormat back fails
that one alone, with "expected:<0[.]05> but was:<0[,]05>"
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M tests/query_test/test_hbo.py
8 files changed, 232 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/24693/9
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 9
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 5: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13905/
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Aug 2026 10:46:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 9:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/24693/9//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24693/9//COMMIT_MSG@47
PS9, Line 47: that one alone, with "expected:<0[.]05> but was:<0[,]05>"
> nit: we usually don't mention the difference with previous patch sets. In t
Rewritten in PS10 so the line says what the test catches, without referring to earlier patch sets.
http://gerrit.cloudera.org:8080/#/c/24693/9/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/9/fe/src/main/java/org/apache/impala/planner/PlanNode.java@582
PS9, Line 582: Locale.ROOT, "%.2f", (double) cardinality / cardinalityBeforeHbo);
> This is a general method that other codes can reuse. Can we move this forma
Moved to PrintUtils.printTwoDecimalsRatio in PS10, test with it in PrintUtilsTest. Kept DecimalFormat rather than String.format - PrintUtils says twice to avoid the latter for IMPALA-1572 - with Locale.ROOT symbols so the value stays parseable, and a new instance per call since DecimalFormat isn't thread-safe.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 9
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2026 19:01:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13949/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 9
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2026 09:13:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23847/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 9
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2026 09:37:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 7: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Aug 2026 16:28:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java@491
PS2, Line 491: .append("\n");
> This adds one line for each PlanNode, which seems too verbose for the defau
PS6: two assertions in test_hbo.py still expected the PS4 text, "(from HBO EXPR_REWRITE)", which PS5 had already reverted here. Running the suite on a minicluster is what surfaced them - they could never have passed. Fixed now, so the tests match the annotation this thread settled on.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Aug 2026 07:54:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23737/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 6
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Aug 2026 08:16:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24693/4/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/4/fe/src/main/java/org/apache/impala/planner/PlanNode.java@490
PS4, Line 490: .append(PrintUtils.printEstCardinality(cardinalityBeforeHbo_))
> Two decimals looks good to me. Thanks!
One thing did change there in PS9. The ratio is formatted under Locale.ROOT now rather than through DecimalFormat's default, so a coordinator whose JVM default is something like de_DE writes 0.05 and not 0,05 - it is the one field on that line meant to be read by a tool. The cardinalities beside it still go through PrintUtils and follow the default locale, like every other estimate in a plan. HboKeyStringTest::testHboRatioIsLocaleIndependent pins it; putting the DecimalFormat back fails that test alone.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 4
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2026 09:13:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 8:
PS8 goes the explain_level route.
Out with the option go its thrift and backend plumbing, Planner.profileExplainDiffers(), Planner.getProfileExplainString() and the second render in Frontend. The details line shows at EXTENDED and above and carries the ratio. Frontend.java and Planner.java are no longer in the patch at all - it is seven files now.
Checked on a cluster rather than by reading: TestHBO 20/20 including test_matching_provenance, HistoricalStatsTest 8/8, and by hand - the line in the profile of an ordinary query with nothing set, absent from EXPLAIN at the default level, present at explain_level=2. I also broke it twice on purpose to see the test catch it: gating at STANDARD fails the assertion that the default EXPLAIN carries no details line, and dropping the ratio fails the profile match. Full precommit is running.
One thing this patch does not answer, worth naming rather than leaving implied: provenance says the estimate moved, not whether the plan moved. isInvertedJoinCheaper() weighs the two children by cardinality, average row size and node count, so an HBO correction on the build side can flip its verdict - and since the join count and the broadcast/partitioned mix stay the same, a plan comparison by operator counts calls that "no change". Filed as IMPALA-15312 instead of growing this change.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 8
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Aug 2026 09:51:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 10: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 10
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Aug 2026 03:06:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24693/4/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24693/4/tests/query_test/test_hbo.py@106
PS4, Line 106: r'HBO match: hash=[0-9a-f]{32}, original estimate=(?!unavailable)\S+',
> Yes, when the planner had no estimate of its own: printEstCardinality() ren
Unchanged in PS11: the negative lookahead still pins that the exact-match case does have an original estimate.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 4
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2026 15:04:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 12: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 12
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Sep 2026 01:15:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 9:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/24693/9//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24693/9//COMMIT_MSG@47
PS9, Line 47: that one alone, with "expected:<0[.]05> but was:<0[,]05>"
nit: we usually don't mention the difference with previous patch sets. In the git history, people read this and would think it's talking about comparison with previous commits.
http://gerrit.cloudera.org:8080/#/c/24693/4/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/4/fe/src/main/java/org/apache/impala/planner/PlanNode.java@490
PS4, Line 490: if (hboMatch_ != null
> One thing did change there in PS9. The ratio is formatted under Locale.ROOT
Nice catch!
http://gerrit.cloudera.org:8080/#/c/24693/9/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/9/fe/src/main/java/org/apache/impala/planner/PlanNode.java@582
PS9, Line 582: Locale.ROOT, "%.2f", (double) cardinality / cardinalityBeforeHbo);
This is a general method that other codes can reuse. Can we move this formatHboRatio method into PrintUtils and use a general name like printTwoDecimalsRatio?
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 9
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2026 14:21:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 5:
> I could not run test_matching_provenance - no cluster here - so the profile behaviour is verified by reading and compiling rather than by running.
Please make sure the modified tests passed before submitting patches for review. If you have limited resources to launch the cluster, you can run the minimal set of services for your testing:
testdata/bin/run-mini-dfs.sh && testdata/bin/run-hive-server.sh && bin/start-impala-cluster.py
The other issue is please reply under the original comments instead of creating new threads to reply which makes it hard to track discussions.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Aug 2026 01:07:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 7:
Loaded TPC-H after all: testdata/datasets/tpch/preload generates it with the
toolchain's dbgen, so "not in my local data set" was wrong rather than
unavoidable. With lineitem and orders present, test_cancellation and
test_runtime_filter_annotation_with_hbo_cardinality both pass, so the caveat in
my last comment is withdrawn - TestHBO is 20 of 20.
Running them turned up something a single run does not show:
test_matching_provenance was not re-runnable. It passes on a fresh HBO history,
fails on a second run against the same cluster, and passes again after a
restart, measured in that order. The aggressive year=2010 query is recorded by
its own first execution, so the second run matches it exactly through
EXPR_REWRITE instead of falling through to IGNORE_PARTITION_CONSTANTS, and the
"partition constants ignored" assertion fails. CI would not see this because
every run gets a fresh cluster; anyone re-running the test locally would.
PS7 gives the string literal a per-run suffix so a run cannot match itself.
Three consecutive full-suite runs against one cluster now pass, 20 of 20 each,
with no restart in between. HistoricalStatsTest is 8/8.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Aug 2026 08:14:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13907/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Aug 2026 11:39:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13905/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Aug 2026 05:54:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23738/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Aug 2026 08:37:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 9: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 9
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2026 13:45:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 7:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
File fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java:
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java@68
PS7, Line 68: carries none
> nit: isn't it EXPR_REWRITE?
Yes - naming it now instead of describing it.
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java@69
PS7, Line 69: data
> nit: IMO, the strategy controls the HBO key string which matches queries, e
You're right, and the sentence had the strategy standing in for the data. Reworked: EXPR_REWRITE matches an equivalent query and so needs no caveat, IGNORE_PARTITION_CONSTANTS can match one that differs in its partition equality predicates - and sizes are the similar-run check's job, not the key's.
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/service/Frontend.java@2145
PS7, Line 2145: : explainStr);
> This looks awkward.. To simplify the code, we can always show the HBO match
Nice that you remembered the tool and had it in mind here :) That is worth measuring rather than guessing at, so I did.
It never runs EXPLAIN itself: the plan reaches it as text someone pasted out of impala-shell, at whatever level they typed. So a query option and a level gate look the same from there.
I ran its parser over a real EXTENDED plan with the annotation on every node. The details line costs it nothing - same cardinalities, same nodes. What does cost it is the "(from HBO)" suffix on the "tuple-ids=... cardinality=..." line, since it matches that line whole. That text is already in master, so it is a fix on my side either way, and a one-line one.
My PS2 reason for the option doesn't hold any more now that the details are on their own line: test_hbo.py doesn't set explain_level, so hbo-*.test stays at STANDARD, and no planner test golden has an HBO match.
So PS8 goes your way. One thing it changes - an EXPLAIN statement at the default level no longer carries the details in its profile, only ordinary queries do, since their plan renders at EXTENDED. That reads fine to me, but tell me if you'd rather keep the profile carrying it in both cases.
http://gerrit.cloudera.org:8080/#/c/24693/7/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24693/7/tests/query_test/test_hbo.py@99
PS7, Line 99: 'explain_level': 2
> The query is not an EXPLAIN. I think we don't need to set explain_level her
Dropped, here and in the aggressive case below. A non-EXPLAIN statement renders its plan at EXTENDED anyway, so the profile has the line without being asked.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Aug 2026 09:50:43 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 8: Code-Review+1
(3 comments)
Thanks for filing the follow-up JIRA! The current patch LGTM. Just have a minor comment.
http://gerrit.cloudera.org:8080/#/c/24693/4/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/4/fe/src/main/java/org/apache/impala/planner/PlanNode.java@490
PS4, Line 490: && detailLevel.ordinal() >= TExplainLevel.EXTENDED.ordinal()) {
> Ratio then. PS8 prints "original estimate=7.30K, ratio=0.05" - two decimals
Two decimals looks good to me. Thanks!
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/service/Frontend.java@2145
PS7, Line 2145: return result;
> Nice that you remembered the tool and had it in mind here :) That is worth
Fine to me as well. Thanks for trying the query option way!
http://gerrit.cloudera.org:8080/#/c/24693/8/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24693/8/tests/query_test/test_hbo.py@302
PS8, Line 302: in '\n'.join(res.data), '\n'.join(res.data)
nit: don't need to reformat this.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 8
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2026 08:52:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24693/8/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24693/8/tests/query_test/test_hbo.py@302
PS8, Line 302: in '\n'.join(res.data), '\n'.join(res.data)
> nit: don't need to reformat this.
Reverted in PS9 - the line is back to what it was.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 8
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2026 09:13:24 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23616/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 1
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Aug 2026 10:03:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 6:
Took the advice and built a minicluster. It paid for itself on the first run:
two of the assertions I had added could never have passed.
test_matching_provenance passed {'explain_level': 2} and
{'enable_explain_hbo': True} as per-query options, and
ImpalaTestSuite.__execute_query calls set_configuration(), which replaces the
configuration instead of merging into it. So those queries ran without
USE_HBO_STATS and STORE_HBO_STATS: no match was recorded, and the profile had
no "HBO match:" line to assert on. All three call sites now use
{**QUERY_OPTIONS, ...}.
test_effective_runtime_filters and
test_runtime_filter_annotation_with_hbo_cardinality expected
"(from HBO EXPR_REWRITE)". getExplainCaveat() returns "" for EXPR_REWRITE by
design, so an expression-rewrite match reads "(from HBO)". Both assertions are
corrected.
While I was in there: a javadoc block had ended up above
profileExplainDiffers() instead of getProfileExplainString(), and the new
QUERY_OPT_FN entry was split over two lines although it fits in 83 columns -
every other split entry in that table actually needs the split.
TestHBO is 18 passed, 2 failed. Both failures are "Could not resolve table
reference" for tpch.lineitem and tpch_parquet.lineitem, which my local data set
does not have, so test_cancellation and
test_runtime_filter_annotation_with_hbo_cardinality do not run here. The
corrected string in the latter therefore rests on the dry-run rather than on
me. HistoricalStatsTest is 8/8.
The accuracy rate, the unavailable case and the EXPLAIN wording are still yours
to call - I have not touched them, and replies on those go in their own threads
from here on.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 6
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Aug 2026 07:55:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 4:
(6 comments)
PS4 addresses the two mechanical points. The accuracy rate, the profile-vs-EXPLAIN question and the unavailable case are answered inline rather than changed, since each of them is your call.
http://gerrit.cloudera.org:8080/#/c/24693/4/be/src/service/query-options.h
File be/src/service/query-options.h:
http://gerrit.cloudera.org:8080/#/c/24693/4/be/src/service/query-options.h@54
PS4, Line 54: TImpalaQueryOptions::ENABLE_EXPLAIN_HBO + 1;
Fixed in PS4. The enum runs 0..206 with no gaps, so _TImpalaQueryOptions_VALUES_TO_NAMES.size() is 207 while NUM_QUERY_OPTIONS still said 206 - the DCHECK would have fired on any debug build.
http://gerrit.cloudera.org:8080/#/c/24693/4/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/24693/4/common/thrift/Query.thrift@837
PS4, Line 837: 207: optional bool enable_explain_hbo = false
Dropped in PS4. store_hbo_stats and aggregated_profile above it have no semicolon either; cte_threshold does, and that is the line I copied.
http://gerrit.cloudera.org:8080/#/c/24693/4/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/4/fe/src/main/java/org/apache/impala/planner/PlanNode.java@490
PS4, Line 490: .append(PrintUtils.printEstCardinality(cardinalityBeforeHbo_))
Makes sense for tooling, and I would rather check the shape with you than pick one.
As written the value goes negative once HBO more than doubles the estimate - cardinalityBeforeHbo_=100 against cardinality_=1000 gives (1 - 900/100) * 100 = -800% - and it divides by zero when the original estimate is 0, which is a real value here rather than the -1 sentinel.
A few ways out: clamp at 0 so anything past 2x reads "0%"; or report the ratio cardinality_ / cardinalityBeforeHbo_ instead, which stays finite and says how far HBO moved the estimate in which direction; or keep your formula and special-case 0 alongside -1 as "N/A". The ratio reads better to me for the low-accuracy nodes you want to find, but the formula is yours - tell me which and I will put it in.
http://gerrit.cloudera.org:8080/#/c/24693/4/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24693/4/tests/query_test/test_hbo.py@67
PS4, Line 67: if 'cardinality' in line]
Leaving the goldens on the cardinality lines then.
http://gerrit.cloudera.org:8080/#/c/24693/4/tests/query_test/test_hbo.py@92
PS4, Line 92: exact_query, {'explain_level': 2, 'enable_explain_hbo': True})
They affect both, and that is worth fixing if you would rather the profile always carried the details.
The profile and EXPLAIN share PlanNode.getExplainString(), and the option is checked inside it, so the line is gated in both. Detail level cannot separate them: the profile asks for VERBOSE, but so does EXPLAIN with explain_level=3, and gating on that would put the line into the EXPLAIN text the option exists to keep stable.
So it needs a signal from the caller - an explicit flag on the profile path, or reading enable_explain_hbo only when the string is built for EXPLAIN. Happy to do that in the next patch set if you would like the profile unconditional.
http://gerrit.cloudera.org:8080/#/c/24693/4/tests/query_test/test_hbo.py@106
PS4, Line 106: r'HBO match: hash=[0-9a-f]{32}, original estimate=(?!unavailable)\S+',
Yes, when the planner had no estimate of its own: printEstCardinality() renders -1 as "unavailable", and cardinalityBeforeHbo_ stays -1 when nothing was computed before the HBO lookup. The negative lookahead pins that the exact-match case does have one, so a regression that lost the original estimate would not pass silently.
That is the same case your "N/A" suggestion covers, so if the accuracy rate lands this assertion should probably move to it.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 4
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sun, 23 Aug 2026 10:33:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24693/5/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/5/fe/src/main/java/org/apache/impala/planner/PlanNode.java@491
PS5, Line 491: expBuilder.append(detailPrefix)
> My answer here on PS2 was wrong and I should not have marked it done. The o
Still how it works in PS11 - the strategy is in the details line, and the annotation carries only the caveat.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2026 15:04:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24693
to look at the new patch set (#11).
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Record the match on the PlanNode and add a details line next to the
cardinality naming the canonicalization strategy that matched, the
opaque hash key, the estimate the planner had computed, and the ratio
between the two, so a tool reading the line does not have to parse
"7.30K" back into a number. The line shows at EXTENDED and above. That
keeps the default EXPLAIN text from growing a line per plan node, and
still leaves the details in the profile of an ordinary query:
statements other than EXPLAIN render their plan at EXTENDED, and the
profile is what remains once the query is gone.
The line is text rather than fields on TPlanNode next to the hbo_hash_keys
that already travel there. Those keys are written when store_hbo_stats is
on, to record a run; this says what a lookup found while planning, and it
is read where the estimate itself is read - in the plan, in the profile,
and in whatever a person pastes into a ticket. A structured form of it is
worth having the day something other than a reader wants it.
Not every match deserves the same trust, and that belongs next to the
number rather than an explain level away. IGNORE_PARTITION_CONSTANTS
drops the constants from partition equality predicates and assumes all
partitions look alike, so the run it matched may have been recorded for
a different partition than the one being planned. Such a match now reads
"(from HBO, key ignores partition constants)". An EXPR_REWRITE match
keeps its constants and describes the query being planned, so it stays
"(from HBO)" and the default EXPLAIN text is unchanged there.
The caveat says what the key leaves out, not what the node does, because
the key covers the subtree: an aggregate over a scan on partition
predicates carries it while having no partition predicates of its own,
which is what the goldens here show.
The ratio goes through a new PrintUtils.printTwoDecimalsRatio, which
formats under Locale.ROOT. It is there for a tool to read rather than
for the eye, and a coordinator whose JVM default is, say, de_DE would
write 0,05 instead. The cardinalities beside it use the default locale
like every other estimate in a plan, so this is about the one field
that claims to be machine-readable, not about the line as a whole.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- PrintUtilsTest, HboKeyStringTest, HistoricalStatsTest: 28 tests, 0
failures. Formatting the ratio under the default locale instead fails
testPrintTwoDecimalsRatio alone, with "expected:<0[.]05> but
was:<0[,]05>"
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/common/PrintUtils.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M fe/src/test/java/org/apache/impala/util/PrintUtilsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M tests/query_test/test_hbo.py
9 files changed, 239 insertions(+), 30 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/24693/11
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 11
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23873/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 11
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2026 15:20:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 7:
(5 comments)
Thanks for updating the patch! The new annotation looks good to me.
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
File fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java:
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java@68
PS7, Line 68: carries none
nit: isn't it EXPR_REWRITE?
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java@69
PS7, Line 69: data
nit: IMO, the strategy controls the HBO key string which matches queries, e.g. EXPR_REWRITE matches equivalent queries, IGNORE_PARTITION_CONSTANTS matches queries that only differ in partition equality predicates.
Data (sizes) matching is done in finding similar runs in the HBO value lists where each run has the input data stats. E.g. if the same query runs twice but in the second run the underlying table/partition is overwritten with double-size data, the HBO key of EXPR_REWRITE matches but the input stats of runs won't match.
In short, HBO key string (and canonicalization strategy) matches queries, and historical runs similarity matches data sizes. Here we can comment that strategy matches equivalent queries don't need caveats.
http://gerrit.cloudera.org:8080/#/c/24693/4/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/4/fe/src/main/java/org/apache/impala/planner/PlanNode.java@490
PS4, Line 490: if (hboMatch_ != null && queryOptions.isEnable_explain_hbo()) {
> Makes sense for tooling, and I would rather check the shape with you than p
I'd prefer cardinality_ / cardinalityBeforeHbo_ for simplicity.
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/service/Frontend.java@2145
PS7, Line 2145: : explainStr);
This looks awkward.. To simplify the code, we can always show the HBO matching details when explain_level >= 2. Will that break your tool for example? https://github.com/alexandrefimov/Query-Doctor
http://gerrit.cloudera.org:8080/#/c/24693/7/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24693/7/tests/query_test/test_hbo.py@99
PS7, Line 99: 'explain_level': 2
The query is not an EXPLAIN. I think we don't need to set explain_level here. Then we can use "self.execute_query(exact_query)" directly since self.execute_query() uses self.client which already has these options set on L82.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Aug 2026 05:18:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24693
to look at the new patch set (#8).
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Record the match on the PlanNode and add a details line next to the
cardinality naming the canonicalization strategy that matched, the
opaque hash key, the estimate the planner had computed, and the ratio
between the two, so a tool reading the line does not have to parse
"7.30K" back into a number. The line shows at EXTENDED and above. That
keeps the default EXPLAIN text from growing a line per plan node, and
still leaves the details in the profile of an ordinary query:
statements other than EXPLAIN render their plan at EXTENDED, and the
profile is what remains once the query is gone.
Not every match deserves the same trust, and that belongs next to the
number rather than an explain level away. IGNORE_PARTITION_CONSTANTS
drops the constants from partition equality predicates and assumes all
partitions look alike, so the run it matched may have been recorded for
a different partition than the one being planned. Such a match now reads
"(from HBO, partition constants ignored)". An EXPR_REWRITE match keeps
its constants and describes the query being planned, so it stays
"(from HBO)" and the default EXPLAIN text is unchanged there.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- HistoricalStatsTest
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M tests/query_test/test_hbo.py
7 files changed, 205 insertions(+), 32 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/24693/8
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 8
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 2:
(7 comments)
Thanks for the review. PS3 addresses all of it; the one I answered rather than changed is the goldens question, below.
http://gerrit.cloudera.org:8080/#/c/24693/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24693/2//COMMIT_MSG@15
PS2, Line 15: IMPALA-15127, including constant-only UNION plans.
> This seems related to a rebase and can be removed.
Done - dropped, and rewrote the message now that IMPALA-15127 is in master.
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java@487
PS2, Line 487: .append("HBO match: strategy=").append(hboStrategy_)
> nit: this seems redundant since the "from HBO" annotation already shows it.
Done - the details line now carries only the hash and the original estimate.
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java@491
PS2, Line 491: .append("\n");
> This adds one line for each PlanNode, which seems too verbose for the defau
Went with the query option: ENABLE_EXPLAIN_HBO, off by default, so the default EXPLAIN text stays as it is. Gating on EXTENDED would have meant moving test_hbo.py to that level and regenerating every golden with resource estimates for the sake of three lines.
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java@1122
PS2, Line 1122: hboHashKey_ = null;
> nit: maybe encapsulate these into a record like PlanNodeStatsMatch. hasHboC
Done - PlanNode keeps a single hboMatch_ record now, hasHboCard_ is gone and the annotation checks it for null.
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/service/HistoricalStats.java
File fe/src/main/java/org/apache/impala/service/HistoricalStats.java:
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/service/HistoricalStats.java@40
PS2, Line 40: public static final class PlanNodeStatsMatch {
> nit: use record in Java 17
Done. HistoricalStatsTest moved to the record accessors along with it.
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/service/HistoricalStats.java@185
PS2, Line 185: * Retrieves the number of output rows from historical stats, trying multiple hash keys
> nit: this also need an update
Done.
http://gerrit.cloudera.org:8080/#/c/24693/2/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24693/2/tests/query_test/test_hbo.py@67
PS2, Line 67: if 'cardinality' in line]
> Do you think we should relax this constraint to also check the HBO details
The details line carries the hash key, which changes whenever the key format changes, so folding it into the golden comparison would mean regenerating every hbo-*.test on each canonicalization tweak. That is why the provenance is checked in test_matching_provenance instead, where the hash is matched by shape rather than by value, and the goldens keep comparing cardinality lines only. Happy to widen the filter if you would rather have it there.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 21 Aug 2026 07:38:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 10:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13954/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 10
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2026 22:35:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java@491
PS2, Line 491: .append("\n");
> PS6: two assertions in test_hbo.py still expected the PS4 text, "(from HBO
Unchanged through PS11: the annotation stays "(from HBO...)" and the details line only shows at EXTENDED and above.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2026 15:04:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 12:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13967/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 12
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2026 20:02:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23722/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sun, 23 Aug 2026 12:11:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 3:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/24693/3/be/src/service/query-options.h
File be/src/service/query-options.h:
http://gerrit.cloudera.org:8080/#/c/24693/3/be/src/service/query-options.h@54
PS3, Line 54: CTE_THRESHOLD
This should be ENABLE_EXPLAIN_HBO now.
http://gerrit.cloudera.org:8080/#/c/24693/3/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/24693/3/common/thrift/Query.thrift@837
PS3, Line 837: ;
nit: don't need semicomma
http://gerrit.cloudera.org:8080/#/c/24693/3/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/3/fe/src/main/java/org/apache/impala/planner/PlanNode.java@490
PS3, Line 490: .append(PrintUtils.printEstCardinality(cardinalityBeforeHbo_))
Showing the accuracy rate would be helpful for external tools so they don't need to parse the human-readable numbers like "7.30K".
(1 - |cardinality_ - cardinalityBeforeHbo_| / cardinalityBeforeHbo_) * 100
We will be interesting in nodes with low accuracy rates which means HBO corrects the estimate a lot. If cardinalityBeforeHbo_ is -1, we can show "N/A". Otherwise, show a string like "85%".
http://gerrit.cloudera.org:8080/#/c/24693/2/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24693/2/tests/query_test/test_hbo.py@63
PS2, Line 63: # To avoid the test being fragile, we only compare the cardinality lines.
: actual_cardinality_lines = [line for line in actual_plan_lines
: if 'cardinality' in line]
: expected_cardinality_lines = [line for line in expected_plan_lines
: if 'cardinality' in line]
> The details line carries the hash key, which changes whenever the key forma
Fine to me. The original cardinality could also change.
http://gerrit.cloudera.org:8080/#/c/24693/3/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24693/3/tests/query_test/test_hbo.py@92
PS3, Line 92: {'explain_level': 2, 'enable_explain_hbo': True}
I thought these just impact EXPLAIN outputs. Do they also impact the plan shown in query profile? I think the plan in the query profile should always be the most verbose version, i.e. with HBO details lines, without setting any of these query options.
http://gerrit.cloudera.org:8080/#/c/24693/3/tests/query_test/test_hbo.py@106
PS3, Line 106: ?!unavailable
Will the original estimate be unavailable?
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sun, 23 Aug 2026 02:34:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24693
to look at the new patch set (#4).
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Annotate the cardinality with the canonicalization strategy that matched,
and, behind the new ENABLE_EXPLAIN_HBO query option, add a line carrying
the hash key and the estimate the planner had computed. Keeping that line
behind an option leaves the default EXPLAIN text unchanged for tooling
that parses it.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- HistoricalStatsTest
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-collection-scan.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-disjunct-selectivity.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-distinct-agg.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-grouping-set.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-iceberg-scan.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-agg.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-nonpartitioned-no-stats.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-nonpartitioned-stats.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-no-stats.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-union-collection-scan.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-union-consts.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-union-same-tables.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
23 files changed, 377 insertions(+), 238 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/24693/4
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 4
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 5:
Yeah, my bad on both counts.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 Aug 2026 04:39:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23703/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 21 Aug 2026 08:01:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24693
to look at the new patch set (#5).
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Record the match on the PlanNode and add a details line next to the
cardinality naming the canonicalization strategy that matched, the
opaque hash key, and the estimate the planner had computed. The query
profile always carries it: that is the artifact left behind once the
query is gone, and the one profile tools read. EXPLAIN carries it only
under the new ENABLE_EXPLAIN_HBO query option, so the text that tooling
parses does not grow a line per plan node by default.
Not every match deserves the same trust, and that belongs where the
number is rather than behind an option. IGNORE_PARTITION_CONSTANTS drops
the constants from partition equality predicates and assumes all
partitions look alike, so the run it matched may have been recorded for
a different partition than the one being planned. Such a match now reads
"(from HBO, partition constants ignored)". An EXPR_REWRITE match keeps
its constants and describes the query being planned, so it stays
"(from HBO)" and the default EXPLAIN text is unchanged there.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- HistoricalStatsTest
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M tests/query_test/test_hbo.py
13 files changed, 245 insertions(+), 38 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/24693/5
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24693
to look at the new patch set (#7).
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Record the match on the PlanNode and add a details line next to the
cardinality naming the canonicalization strategy that matched, the
opaque hash key, and the estimate the planner had computed. The query
profile always carries it: that is the artifact left behind once the
query is gone, and the one profile tools read. EXPLAIN carries it only
under the new ENABLE_EXPLAIN_HBO query option, so the text that tooling
parses does not grow a line per plan node by default.
Not every match deserves the same trust, and that belongs where the
number is rather than behind an option. IGNORE_PARTITION_CONSTANTS drops
the constants from partition equality predicates and assumes all
partitions look alike, so the run it matched may have been recorded for
a different partition than the one being planned. Such a match now reads
"(from HBO, partition constants ignored)". An EXPR_REWRITE match keeps
its constants and describes the query being planned, so it stays
"(from HBO)" and the default EXPLAIN text is unchanged there.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- HistoricalStatsTest
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M tests/query_test/test_hbo.py
13 files changed, 250 insertions(+), 37 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/24693/7
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24693
to look at the new patch set (#6).
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Record the match on the PlanNode and add a details line next to the
cardinality naming the canonicalization strategy that matched, the
opaque hash key, and the estimate the planner had computed. The query
profile always carries it: that is the artifact left behind once the
query is gone, and the one profile tools read. EXPLAIN carries it only
under the new ENABLE_EXPLAIN_HBO query option, so the text that tooling
parses does not grow a line per plan node by default.
Not every match deserves the same trust, and that belongs where the
number is rather than behind an option. IGNORE_PARTITION_CONSTANTS drops
the constants from partition equality predicates and assumes all
partitions look alike, so the run it matched may have been recorded for
a different partition than the one being planned. Such a match now reads
"(from HBO, partition constants ignored)". An EXPR_REWRITE match keeps
its constants and describes the query being planned, so it stays
"(from HBO)" and the default EXPLAIN text is unchanged there.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- HistoricalStatsTest
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M tests/query_test/test_hbo.py
13 files changed, 245 insertions(+), 37 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/24693/6
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 6
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 8: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 8
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Aug 2026 14:18:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 2:
(7 comments)
The details look pretty good!
http://gerrit.cloudera.org:8080/#/c/24693/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24693/2//COMMIT_MSG@14
PS2, Line 14: Keep the empty-scan matching introduced by
: IMPALA-15127, including constant-only UNION plans.
This seems related to a rebase and can be removed.
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java@487
PS2, Line 487: hboStrategy_
nit: this seems redundant since the "from HBO" annotation already shows it.
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java@491
PS2, Line 491: .append("\n");
This adds one line for each PlanNode, which seems too verbose for the default explain_level. It also changes the user-visible EXPLAIN text that external tooling may parse. Can we only add it when explain_level is set to >= 2, i.e. EXTENDED or VERBOSE? Or we can add a new query option like ENABLE_EXPLAIN_HBO (similar to ENABLE_EXPLAIN_CALCITE) to control whether to show these details. A query option seems more friendly for testing these lines.
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/planner/PlanNode.java@1120
PS2, Line 1120: cardinalityBeforeHbo_ = -1;
: hboStrategy_ = null;
: hboHashKey_ = null;
nit: maybe encapsulate these into a record like PlanNodeStatsMatch. hasHboCard_ can then be replaced by checking if this record isn't null.
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/service/HistoricalStats.java
File fe/src/main/java/org/apache/impala/service/HistoricalStats.java:
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/service/HistoricalStats.java@40
PS2, Line 40: public static final class PlanNodeStatsMatch {
nit: use record in Java 17
http://gerrit.cloudera.org:8080/#/c/24693/2/fe/src/main/java/org/apache/impala/service/HistoricalStats.java@185
PS2, Line 185: the number of output rows
nit: this also need an update
http://gerrit.cloudera.org:8080/#/c/24693/2/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24693/2/tests/query_test/test_hbo.py@63
PS2, Line 63: # To avoid the test being fragile, we only compare the cardinality lines.
: actual_cardinality_lines = [line for line in actual_plan_lines
: if 'cardinality' in line]
: expected_cardinality_lines = [line for line in expected_plan_lines
: if 'cardinality' in line]
Do you think we should relax this constraint to also check the HBO details lines?
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 20 Aug 2026 14:11:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24693/4/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24693/4/fe/src/main/java/org/apache/impala/planner/PlanNode.java@490
PS4, Line 490: .append(PrintUtils.printEstCardinality(cardinalityBeforeHbo_))
> I'd prefer cardinality_ / cardinalityBeforeHbo_ for simplicity.
Ratio then. PS8 prints "original estimate=7.30K, ratio=0.05" - two decimals, and "N/A" when the original estimate is unavailable or zero, since zero doesn't divide either, which I missed when I called the ratio finite above. That case is not theoretical: on the minicluster a merge aggregate came out with a planner estimate of 0 and read "ratio=N/A" on the first try. Percentage instead if you prefer it that way.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 4
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Aug 2026 09:51:11 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23828/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 8
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 27 Aug 2026 09:58:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23852/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 10
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2026 19:18:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 11:
PS11 rewords the caveat on the cardinality line: "(from HBO, partition constants ignored)" is now "(from HBO, key ignores partition constants)". The words moved because the caveat travels up the plan with the key - an aggregate over a scan on partition predicates carries it while having no partition predicates of its own, which the goldens here show. The enum returns the words now and PlanNode adds the punctuation around them.
Nothing else moved: same gating, same details line, same ratio. PrintUtilsTest and HistoricalStatsTest pass, and TestHBO passes end to end. Putting the old wording back in the code and leaving the goldens alone fails three of its tests, so they do check this.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 11
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2026 15:05:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/24693/7/fe/src/main/java/org/apache/impala/service/Frontend.java@2145
PS7, Line 2145: : explainStr);
> Fine to me as well. Thanks for trying the query option way!
PS11 keeps the level gate and adds no query option.
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2026 15:04:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24693 )
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
Patch Set 11: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24693/11//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24693/11//COMMIT_MSG@14
PS11, Line 14: Record the match on the PlanNode and add a details line next to the
nit: it'd be helpful to paste an example of a PlanNode in the commit message
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 11
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2026 15:26:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15236: Expose HBO match provenance
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24693
to look at the new patch set (#3).
Change subject: IMPALA-15236: Expose HBO match provenance
......................................................................
IMPALA-15236: Expose HBO match provenance
The plan says "(from HBO)" when a cardinality comes from a historical
run, but not which key matched or what the planner would have estimated
on its own. Investigating a surprising estimate means turning on debug
logging and running the query again.
Annotate the cardinality with the canonicalization strategy that matched,
and, behind the new ENABLE_EXPLAIN_HBO query option, add a line carrying
the hash key and the estimate the planner had computed. Keeping that line
behind an option leaves the default EXPLAIN text unchanged for tooling
that parses it.
Reset the match metadata before each lookup so a later miss cannot show
stale provenance left by an earlier pass over the same node.
Testing:
- HistoricalStatsTest
- TestHBO::test_matching_provenance
Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Assisted-by: claude-opus-5 (Claude Code)
---
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-collection-scan.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-disjunct-selectivity.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-distinct-agg.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-grouping-set.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-iceberg-scan.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-multiple-scans.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-agg.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-nonpartitioned-no-stats.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-nonpartitioned-stats.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-no-stats.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-single-scan-partitioned-stats.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-union-collection-scan.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-union-consts.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-union-same-tables.test
M testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
23 files changed, 376 insertions(+), 237 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/24693/3
--
To view, visit http://gerrit.cloudera.org:8080/24693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6d2deaaf78a2a41353454ba634a247d8c69825bf
Gerrit-Change-Number: 24693
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>