You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2026/07/06 14:07:48 UTC
[Impala-ASF-CR] [WIP] IMPALA-15127: Support HBO for UnionNode cardinality
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24556
Change subject: [WIP] IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
[WIP] IMPALA-15127: Support HBO for UnionNode cardinality
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
4 files changed, 181 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/1
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 17: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13864/
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 17
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Aug 2026 17:40:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@376
PS8, Line 376: sb.append("constOps:").append(constExprLists_.size()).append("|");
> PS13 addes the constants into the key.
The canonicalized constant rows in the key look good. One edge case with the (LiteralExpr) cast - commented on PS13.
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@391
PS8, Line 391: List<PlanNode.HboKeyedNode> sortedChildren = PlanNode.buildSortedHboNodes(
> Done. Added an e2e test.
Thanks - the swapped-predicates pair in hbo-union-same-tables.test is exactly the regression coverage I was after.
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@415
PS8, Line 415: populateHboThriftFields(msg, serialCtx);
> Extended the code to allow empty scan input stats for matching const-only U
Verified PS13: both preconditions are gone, scanInputStatsMatch() matches an empty stats list only against another empty one, and hbo-union-consts.test covers the DISTINCT shape (tracked AGGREGATE over an untracked const-only UNION). Write and read paths are consistent now. Thanks!
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/service/HistoricalStats.java
File fe/src/main/java/org/apache/impala/service/HistoricalStats.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/service/HistoricalStats.java@118
PS8, Line 118: for (int i = 0; i < runs.size(); i++) {
> Searching newest-to-oldest still have problems. E.g. when the list has only
Agreed. The [110, 122] example shows newest-first has shadowing cases of its own, and since the write path replaces similar runs, the list stays self-correcting over time. Keeping first-match is fine.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Aug 2026 07:34:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 16: Code-Review+2
(1 comment)
Carrying Aman's +2 after updating the commit message. Thanks all for your reviews!
http://gerrit.cloudera.org:8080/#/c/24556/15//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24556/15//COMMIT_MSG@43
PS15, Line 43: UnionNod
> nit: you mean UnionNode
Done
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 16
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Aug 2026 12:52:16 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 10:
(5 comments)
Thanks for the comments! They are quite helpful!
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@376
PS8, Line 376: StringBuilder sb = new StringBuilder(statsType.name()).append(":UnionNode:");
> But I think I disagree with that.
I think it's still better than nothing. E.g. the following two queries only differ in the number of constant union operands. The JoinNode cardinalities differ a lot. Using different HBO keys for the UnionNode helps to have different keys for the JoinNode.
select count(*) from functional.alltypes a inner join (
select distinct int_col from functional.alltypestiny
union all
values (2), (3), (4)
) b on a.int_col = b.int_col;
select count(*) from functional.alltypes a inner join (
select distinct int_col from functional.alltypestiny
union all
values (2), (3)
) b on a.int_col = b.int_col;
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@383
PS8, Line 383: if (i > 0) sb.append(",");
> Should we include `limit_` in this key, as `HdfsScanNode` does? A top-level
Nice catch! LIMIT is also missing in AggregationNode.
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@391
PS8, Line 391: public void appendScanInputStats(TPlanNodeRun execStats) {
> I don't think sorting this list by `EXPR_REWRITE` keeps it aligned with eve
That's a good example! Initially I thought the IGNORE_PARTITION_CONSTANTS strategy assumes that partitions in the same table have similar stats. So it's OK to let these two queries match. But if the partitions have different numbers of input rows and if int_col=1 and int_col=-999 have different selectivity, matching these two queries gives a wrong cardinality.
BTW, I think the planner should merge the union operands into one if they are simple scan on the same table, though we don't have such optimization yet.
I'll think more about this and address this in the next patch set.
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@415
PS8, Line 415: msg.union_node = new TUnionNode(
> `populateHboThriftFields()` also runs for a constants-only Union. Its key i
Nice catch! We shouldn't write HBO stats for such UnionNodes.
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/service/HistoricalStats.java
File fe/src/main/java/org/apache/impala/service/HistoricalStats.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/service/HistoricalStats.java@118
PS8, Line 118: for (int i = 0; i < runs.size(); i++) {
> Similarity ranges can overlap, so returning the first match does not always
I think this is the limitaion of simple threshold matching. For this example, keeping 120 can help more lookups, e.g. for 130, so it's still useful. The run of 120 might not be a stale stats. It could belong to another kind of partitions. So I think keeping it still helps.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Aug 2026 12:57:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Aleksandr Efimov, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24556
to look at the new patch set (#12).
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
HBO Key String
--------------
The HBO key string of a UnionNode consits of the number of const
operands, the optional LIMIT, and the key strings of its children.
To make the HBO key string consistent for different operand orders,
the children are sorted by their concatenated and sorted scan table
names. E.g. an operand with only one table scan in its subtree uses the
table name as the sort key, an operand with multiple table scans uses a
sort key like "tbl1,tbl2,tbl3".
In case when two operands have the same set of table scans, we use the
original order in query for simplicity. Another query that flips their
order will have a different HBO key, which misses the HBO stats. Such
cases are rare in practice, so we keep them unoptimized.
Historical Runs Matching
------------------------
If the HBO key strings match, we pick one from the list of historical
runs corresponding to that key.
UnionNode is the first HBO-supported node type that has multiple
children so we need to extend the existing matching mechanism to
supports matching multiple scan nodes.
Each UnionNode collects all the leaf scan input stats inside its subtree
in pre-order traversal. The order of visiting its children is the same
as the operand order used in the HBO key string.
While matching a historical run, element-wise comparison of all
TScanInputStats entries is performed to ensure they are similar
respectively. Take the following JoinNode runs as an example. The
children (sorted) are ScanNodes A, B, C.
UnionNode UnionNode
/ | \ / | \
A B C A' B' C'
The matching requires similar(A, A') && similar(B, B') && similar(C, C').
Similarity of a scan node pair is checked in the following way:
- If both sides have valid input_rows (come from HMS numRows stats),
compare row count.
- If either input_rows is missing,
- For EXPR_REWRITE, an exact catalog version match means similar.
if catalog versions differ, compare input file sizes.
- For other strategies, catalog version is not a reliable signal,
so compare input file sizes only.
Note that the hash key matching already ensures table names and
conjuncts are matched. It's done before finding the historical run so we
don't need to compare table names or conjuncts here.
Testing
- Added FE tests for HBO key strings.
- Added FE tests to matching historical runs.
- Added e2e tests on all UNION types.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-collection-scan.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-same-tables.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
12 files changed, 727 insertions(+), 86 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/12
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 12
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 11:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24556/11/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/11/fe/src/main/java/org/apache/impala/planner/PlanNode.java@50
PS11, Line 50: import org.apache.impala.common.Pair;
unused import: org.apache.impala.common.Pair
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Aug 2026 15:00:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 13:
(1 comment)
Checked PS13: the empty-scan-stats handling is consistent end to end now; resolved those threads. Found one new issue in the constRows key generation, see the inline comment.
http://gerrit.cloudera.org:8080/#/c/24556/13/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/13/fe/src/main/java/org/apache/impala/planner/UnionNode.java@381
PS13, Line 381: .map(c -> ExprCanonicalizer.canonicalizeLiteral((LiteralExpr) c))
constExprLists_ isn't guaranteed to contain only LiteralExprs, so this cast can throw ClassCastException and fail planning when HBO is enabled.
createUnionPlan() adds the result exprs of any no-FROM operand without a constness check, and FoldConstantsRule skips non-constant exprs, so `select rand() union all select id from functional.alltypes` puts a FunctionCallExpr into constExprLists_. Generating this union's key (its own populateHboThriftFields(), or an ancestor's key via the operand list) then throws. With enable_expr_rewrites=false even `select 1 + 1 union all select 2` hits it, since nothing folds 1+1 into a literal.
Could we check instanceof and return null for non-literal rows (skip HBO for the subtree), or render them with toSql()? A regression test with a nondeterministic operand like rand() would pin this down.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 13
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Aug 2026 07:35:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24556
to look at the new patch set (#5).
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
The HBO key string of a UnionNode consits of the number of const
operands and the key strings of its children. The children strings are
sorted so the operand order doesn't matter.
Testing
- Added FE tests for HBO key strings.
- Added e2e tests on all UNION types.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
5 files changed, 234 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/5
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@376
PS8, Line 376: sb.append("constOps:").append(constExprLists_.size()).append("|");
So we're just counting the constant operations and not showing what's in them. Is it possible that a PlanNode above this UnionNode might specifically have a filter pointing to a constant and the cardinality changes if the constant is different?
Maybe this can be ignored? These are estimates, after all. But just wanted to throw this out there.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Jul 2026 15:40:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@415
PS8, Line 415: populateHboThriftFields(msg, serialCtx);
> `populateHboThriftFields()` also runs for a constants-only Union. Its key i
PS12 still leaves a no-scan path through a parent node. `select 1 union select 2` creates an AggregationNode above the constants-only UnionNode. The Union skips HBO serialization, but the aggregation still gets an HBO key while `appendScanInputStats()` leaves `scan_input_stats` unset. A later update for the same key reaches `HistoricalStats.getSimilarRunIndex()` and fails its non-empty precondition. Could we handle the empty scan list in the shared `populateHboThriftFields()` path, or otherwise suppress HBO fields for the parent, and add a regression test for this DISTINCT plan shape?
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Aug 2026 07:36:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Aleksandr Efimov, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24556
to look at the new patch set (#11).
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
HBO Key String
--------------
The HBO key string of a UnionNode consits of the number of const
operands, the optional LIMIT, and the key strings of its children.
To make the HBO key string consistent for different operand orders,
the children are sorted by their concatenated and sorted scan table
names. E.g. an operand with only one table scan in its subtree uses the
table name as the sort key, an operand with multiple table scans uses a
sort key like "tbl1,tbl2,tbl3".
In case when two operands have the same set of table scans, we use the
original order in query for simplicity. Another query that flips their
order will have a different HBO key, which misses the HBO stats. Such
cases are rare in practice, so we keep them unoptimized.
Historical Runs Matching
------------------------
If the HBO key strings match, we pick one from the list of historical
runs corresponding to that key.
UnionNode is the first HBO-supported node type that has multiple
children so we need to extend the existing matching mechanism to
supports matching multiple scan nodes.
Each UnionNode collects all the leaf scan input stats inside its subtree
in pre-order traversal. The order of visiting its children is the same
as the operand order used in the HBO key string.
While matching a historical run, element-wise comparison of all
TScanInputStats entries is performed to ensure they are similar
respectively. Take the following JoinNode runs as an example. The
children (sorted) are ScanNodes A, B, C.
UnionNode UnionNode
/ | \ / | \
A B C A' B' C'
The matching requires similar(A, A') && similar(B, B') && similar(C, C').
Similarity of a scan node pair is checked in the following way:
- If both sides have valid input_rows (come from HMS numRows stats),
compare row count.
- If either input_rows is missing,
- For EXPR_REWRITE, an exact catalog version match means similar.
if catalog versions differ, compare input file sizes.
- For other strategies, catalog version is not a reliable signal,
so compare input file sizes only.
Note that the hash key matching already ensures table names and
conjuncts are matched. It's done before finding the historical run so we
don't need to compare table names or conjuncts here.
Testing
- Added FE tests for HBO key strings.
- Added FE tests to matching historical runs.
- Added e2e tests on all UNION types.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-same-tables.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
11 files changed, 667 insertions(+), 86 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/11
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Aleksandr Efimov, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24556
to look at the new patch set (#16).
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
HBO Key String
--------------
The HBO key string of a UnionNode consits of the optional LIMIT, the
canonicalized const operands, and the key strings of its children.
To make the HBO key string consistent for different operand orders,
the children are sorted by their concatenated and sorted scan table
names. E.g. an operand with only one table scan in its subtree uses the
table name as the sort key, an operand with multiple table scans uses a
sort key like "tbl1,tbl2,tbl3".
In case when two operands have the same set of table scans, we use the
original order in query for simplicity. Another query that flips their
order will have a different HBO key, which misses the HBO stats. Such
cases are rare in practice, so we keep them unoptimized.
Historical Runs Matching
------------------------
If the HBO key strings match, we pick one from the list of historical
runs corresponding to that key.
UnionNode is the first HBO-supported node type that has multiple
children so we need to extend the existing matching mechanism to
supports matching multiple scan nodes.
Each UnionNode collects all the leaf scan input stats inside its subtree
in pre-order traversal. The order of visiting its children is the same
as the operand order used in the HBO key string.
While matching a historical run, element-wise comparison of all
TScanInputStats entries is performed to ensure they are similar
respectively. Take the following UnionNode runs as an example. The
children (sorted) are ScanNodes A, B, C.
UnionNode UnionNode
/ | \ / | \
A B C A' B' C'
The matching requires similar(A, A') && similar(B, B') && similar(C, C').
Similarity of a scan node pair is checked in the following way:
- If both sides have valid input_rows (come from HMS numRows stats),
compare row count.
- If either input_rows is missing,
- For EXPR_REWRITE, an exact catalog version match means similar.
if catalog versions differ, compare input file sizes.
- For other strategies, catalog version is not a reliable signal,
so compare input file sizes only.
Note that the hash key matching already ensures table names and
conjuncts are matched. It's done before finding the historical run so we
don't need to compare table names or conjuncts here.
For UnionNodes that have only constant operands, there are no scan input
stats. The corresponding HBO value list will have at most one item since
the HBO key matching already ensures the constants are identical.
Testing
- Added FE tests for HBO key strings.
- Added FE tests to matching historical runs.
- Added e2e tests on all UNION types.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/ExprCanonicalizer.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-collection-scan.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-consts.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-same-tables.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
14 files changed, 854 insertions(+), 95 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/16
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 16
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@376
PS8, Line 376: sb.append("constOps:").append(constExprLists_.size()).append("|");
> Without this, the key string will be too simple for constant VALUES/SELECT
But I think I disagree with that.
If you had a Union node with only constants, the cardinality coming from the union node will be practically insignificant anyway, no? Whether there are 33 constants or 34 constants, it doesn't really affect the upper nodes that much since the caardinality is barely different...or if it does, then we need to solve the original problem of knowing exactly what the constants are.
I'm not gonna make too much of a fuss on this point. I might be missing some nuance, but I think that in the end, it doesn't matter whether we keep it there or not. If you want to leave it, I'm ok with that :)
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sat, 01 Aug 2026 02:23:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 12:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13841/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 12
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Aug 2026 00:43:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23614/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 13
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Aug 2026 06:10:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 12: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 12
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Aug 2026 05:37:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 17:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13864/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 17
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Aug 2026 12:52:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 17:
> Patch Set 17: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13864/
The failure is unrelated to this patch:
ERROR: Could not install packages due to an OSError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl.metadata (Caused by ResponseError('too many 502 error responses'))
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 17
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 20 Aug 2026 01:20:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24556
to look at the new patch set (#7).
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
HBO Key String
--------------
The HBO key string of a UnionNode consits of the number of const
operands and the key strings of its children. The children strings are
sorted so the operand order doesn't matter.
Historical Runs Matching
------------------------
If the HBO key strings match, we pick one from the list of historical
runs corresponding to that key.
UnionNode is the first HBO-supported node type that has multiple
children so we need to extend the existing matching mechanism to
supports matching multiple scan nodes.
Each UnionNode collects all the leaf scan input stats inside its subtree.
These scan input stats are sorted using the same order of the sorted
operands.
While matching a historical run, element-wise comparison of all
TScanInputStats entries is performed to ensure they are similar
respectively. Take the following JoinNode runs as an example.
UnionNode UnionNode
/ | \ / | \
A B C A' B' C'
The matching requires similar(A, A') && similar(B, B') && similar(C, C').
Similarity of a scan node pair is checked in the following way:
- If both sides have valid input_rows (come from HMS numRows stats),
compare row count.
- If either input_rows is missing,
- For EXPR_REWRITE, an exact catalog version match means similar.
if catalog versions differ, compare input file sizes.
- For other strategies, catalog version is not a reliable signal,
so compare input file sizes only.
Note that the hash key matching already ensures table names and
conjuncts are matched. It's done before finding the historical run so we
don't need to compare table names or conjuncts here.
Testing
- Added FE tests for HBO key strings.
- Added FE tests to matching historical runs.
- Added e2e tests on all UNION types.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
8 files changed, 521 insertions(+), 86 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/7
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 7
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23237/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 01:53:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Aleksandr Efimov, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24556
to look at the new patch set (#10).
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
HBO Key String
--------------
The HBO key string of a UnionNode consits of the number of const
operands, the optional limit, and the key strings of its children. The
children strings are sorted so the operand order doesn't matter.
Historical Runs Matching
------------------------
If the HBO key strings match, we pick one from the list of historical
runs corresponding to that key.
UnionNode is the first HBO-supported node type that has multiple
children so we need to extend the existing matching mechanism to
supports matching multiple scan nodes.
Each UnionNode collects all the leaf scan input stats inside its subtree.
These scan input stats are sorted using the same order of the sorted
operands.
While matching a historical run, element-wise comparison of all
TScanInputStats entries is performed to ensure they are similar
respectively. Take the following JoinNode runs as an example.
UnionNode UnionNode
/ | \ / | \
A B C A' B' C'
The matching requires similar(A, A') && similar(B, B') && similar(C, C').
Similarity of a scan node pair is checked in the following way:
- If both sides have valid input_rows (come from HMS numRows stats),
compare row count.
- If either input_rows is missing,
- For EXPR_REWRITE, an exact catalog version match means similar.
if catalog versions differ, compare input file sizes.
- For other strategies, catalog version is not a reliable signal,
so compare input file sizes only.
Note that the hash key matching already ensures table names and
conjuncts are matched. It's done before finding the historical run so we
don't need to compare table names or conjuncts here.
Testing
- Added FE tests for HBO key strings.
- Added FE tests to matching historical runs.
- Added e2e tests on all UNION types.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
9 files changed, 562 insertions(+), 86 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/10
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 8:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@391
PS8, Line 391: List<PlanNode.HboKeyedNode> sortedChildren = PlanNode.buildSortedHboNodes(
> I don't think sorting this list by `EXPR_REWRITE` keeps it aligned with eve
This approach sounds reasonable if the fallback order is used for both the Union key and `TScanInputStats`. Then the duplicate-table example above becomes order-sensitive and does not match. Could you add that Q1/Q2 case as a regression test? It covers the important fallback path.
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@415
PS8, Line 415: populateHboThriftFields(msg, serialCtx);
> Thanks, the PS10 guard looks good. Could you also add the regression test f
Changing the log level would only hide the failed write. Since PS10 fixes this before serialization, I think the focused test should build a constants-only Union and verify that its Thrift plan has no HBO hash keys. An end-to-end test can also rerun the query and check that the Union has no `(from HBO)` annotation, but I would not assert log text.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Aug 2026 20:01:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Aleksandr Efimov, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24556
to look at the new patch set (#13).
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
HBO Key String
--------------
The HBO key string of a UnionNode consits of the optional LIMIT, the
canonicalized const operands, and the key strings of its children.
To make the HBO key string consistent for different operand orders,
the children are sorted by their concatenated and sorted scan table
names. E.g. an operand with only one table scan in its subtree uses the
table name as the sort key, an operand with multiple table scans uses a
sort key like "tbl1,tbl2,tbl3".
In case when two operands have the same set of table scans, we use the
original order in query for simplicity. Another query that flips their
order will have a different HBO key, which misses the HBO stats. Such
cases are rare in practice, so we keep them unoptimized.
Historical Runs Matching
------------------------
If the HBO key strings match, we pick one from the list of historical
runs corresponding to that key.
UnionNode is the first HBO-supported node type that has multiple
children so we need to extend the existing matching mechanism to
supports matching multiple scan nodes.
Each UnionNode collects all the leaf scan input stats inside its subtree
in pre-order traversal. The order of visiting its children is the same
as the operand order used in the HBO key string.
While matching a historical run, element-wise comparison of all
TScanInputStats entries is performed to ensure they are similar
respectively. Take the following JoinNode runs as an example. The
children (sorted) are ScanNodes A, B, C.
UnionNode UnionNode
/ | \ / | \
A B C A' B' C'
The matching requires similar(A, A') && similar(B, B') && similar(C, C').
Similarity of a scan node pair is checked in the following way:
- If both sides have valid input_rows (come from HMS numRows stats),
compare row count.
- If either input_rows is missing,
- For EXPR_REWRITE, an exact catalog version match means similar.
if catalog versions differ, compare input file sizes.
- For other strategies, catalog version is not a reliable signal,
so compare input file sizes only.
Note that the hash key matching already ensures table names and
conjuncts are matched. It's done before finding the historical run so we
don't need to compare table names or conjuncts here.
For UnionNodes that have only constant operands, there are no scan input
stats. The corresponding HBO value list will have at most one item since
the HBO key matching already ensures the constants are identical.
Testing
- Added FE tests for HBO key strings.
- Added FE tests to matching historical runs.
- Added e2e tests on all UNION types.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/ExprCanonicalizer.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-collection-scan.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-consts.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-same-tables.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
14 files changed, 792 insertions(+), 94 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/13
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 13
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 17: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 17
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Aug 2026 12:52:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 5:
> Patch Set 4: Code-Review+2
>
> LGTM. A related thing is about other set operations such as EXCEPT, INTERSECT. Would those appear in a different patch ? I tend to think of all set operations as a unit. Either way is ok.
EXCEPT and INTERSECT are converted to anti/semi joins. I'll add these tests in the JoinNode patch.
Rebased this UnionNode patch to be directly on top of the AggregationNode patch since it got +2. The JoinNode patch is moved to be on top of this patch.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 02:40:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: [WIP] IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 1:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24556/1/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/1/fe/src/main/java/org/apache/impala/planner/UnionNode.java@21
PS1, Line 21: import java.util.Collections;
unused import: java.util.Collections
http://gerrit.cloudera.org:8080/#/c/24556/1/fe/src/main/java/org/apache/impala/planner/UnionNode.java@370
PS1, Line 370: List<PlanNode.HboKeyedNode> sortedChildren = PlanNode.buildSortedHboNodes(statsType, strategy, children_);
line too long (110 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 06 Jul 2026 14:10:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23456/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 7
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jul 2026 10:40:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 10: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@376
PS8, Line 376: StringBuilder sb = new StringBuilder(statsType.name()).append(":UnionNode:");
> I think it's still better than nothing. E.g. the following two queries only
I can't say for sure if it's better than nothing because if the cardinality barely changes by adding N constants versus N+1 constants, then it's a missed opportunity to use HBO stats. A false negative rather than a false positive.
But I won't argue this any further :) I don't think there's a right or wrong answer here.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Aug 2026 13:58:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24556
to look at the new patch set (#2).
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
The HBO key string of a UnionNode consits of the number of const
operands and the key strings of its children. The children strings are
sorted so the operand order doesn't matter.
Testing
- Added FE tests for HBO key strings.
- Added e2e tests on all UNION types.
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
4 files changed, 214 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/2
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24556
to look at the new patch set (#3).
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
The HBO key string of a UnionNode consits of the number of const
operands and the key strings of its children. The children strings are
sorted so the operand order doesn't matter.
Testing
- Added FE tests for HBO key strings.
- Added e2e tests on all UNION types.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
4 files changed, 214 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/3
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23583/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Aug 2026 15:21:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: [WIP] IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23106/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 06 Jul 2026 14:31:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
HBO Key String
--------------
The HBO key string of a UnionNode consits of the optional LIMIT, the
canonicalized const operands, and the key strings of its children.
To make the HBO key string consistent for different operand orders,
the children are sorted by their concatenated and sorted scan table
names. E.g. an operand with only one table scan in its subtree uses the
table name as the sort key, an operand with multiple table scans uses a
sort key like "tbl1,tbl2,tbl3".
In case when two operands have the same set of table scans, we use the
original order in query for simplicity. Another query that flips their
order will have a different HBO key, which misses the HBO stats. Such
cases are rare in practice, so we keep them unoptimized.
Historical Runs Matching
------------------------
If the HBO key strings match, we pick one from the list of historical
runs corresponding to that key.
UnionNode is the first HBO-supported node type that has multiple
children so we need to extend the existing matching mechanism to
supports matching multiple scan nodes.
Each UnionNode collects all the leaf scan input stats inside its subtree
in pre-order traversal. The order of visiting its children is the same
as the operand order used in the HBO key string.
While matching a historical run, element-wise comparison of all
TScanInputStats entries is performed to ensure they are similar
respectively. Take the following UnionNode runs as an example. The
children (sorted) are ScanNodes A, B, C.
UnionNode UnionNode
/ | \ / | \
A B C A' B' C'
The matching requires similar(A, A') && similar(B, B') && similar(C, C').
Similarity of a scan node pair is checked in the following way:
- If both sides have valid input_rows (come from HMS numRows stats),
compare row count.
- If either input_rows is missing,
- For EXPR_REWRITE, an exact catalog version match means similar.
if catalog versions differ, compare input file sizes.
- For other strategies, catalog version is not a reliable signal,
so compare input file sizes only.
Note that the hash key matching already ensures table names and
conjuncts are matched. It's done before finding the historical run so we
don't need to compare table names or conjuncts here.
For UnionNodes that have only constant operands, there are no scan input
stats. The corresponding HBO value list will have at most one item since
the HBO key matching already ensures the constants are identical.
Testing
- Added FE tests for HBO key strings.
- Added FE tests to matching historical runs.
- Added e2e tests on all UNION types.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Reviewed-on: http://gerrit.cloudera.org:8080/24556
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/ExprCanonicalizer.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-collection-scan.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-consts.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-same-tables.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
14 files changed, 854 insertions(+), 95 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 18
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 10:
(2 comments)
Before making the changes which might be large, let me reply the discussion first. We can discuss more until we make a consensus.
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@391
PS8, Line 391: public void appendScanInputStats(TPlanNodeRun execStats) {
> That's a good example! Initially I thought the IGNORE_PARTITION_CONSTANTS s
I'm thinking a strategy-independent order that can be used here and in UnionNode.generateHboKeyString(). Then we don't need to generate a list of TScanInputStats for each strategy and send them to BE. If we do so, we need more lists in the future when HBO supports more stats types - they could also impact the order of the HBO key string since they would add more info in the key.
Here is one approach for soring operands that I found from Presto:
- First, sort by the scan table name string of each operand. If the operand has a single scan in its subtree, use the table name as the string. If the operand has multiple scans, use a comma seperated list of the sorted table names as the string.
- If there are duplicate strings, abort sorting and use the original order of the query. Note that such cases are rare in practise so might not worth efforts in optimization.
Note that Presto adds a flag, use_perfectly_consistent_histories, to sort by the HBO key strings. But it's disabled by default.
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@415
PS8, Line 415: msg.union_node = new TUnionNode(
> Thanks, the PS10 guard looks good. Could you also add the regression test f
The query won't fail since the HBO write path is independent and async to the query execution. There are just INFO level logs like this:
I20260805 20:38:00.834429 135178 impala-server.cc:1797] Storing history stats for query a9431bb921856083:684eef8300000000
I20260805 20:38:00.834708 135178 JniFrontend.java:1001] execution stats from BE: THistoricalStatsUpdate(plan_node_runs:[TPlanNodeRunWithKeys(run:TPlanNodeRun(num_rows:2), hash_keys:{EXPR_REWRITE=a41799e17d0931a61bd649d4fdb57037}, stats_type:CARDINALITY)])
I20260805 20:38:00.835738 135178 jni-util.cc:335] java.lang.IllegalStateException
at com.google.common.base.Preconditions.checkState(Preconditions.java:496)
at org.apache.impala.service.HistoricalStats.getSimilarRunIndex(HistoricalStats.java:116)
at org.apache.impala.service.HistoricalStats.writePlanNodeStats(HistoricalStats.java:147)
at org.apache.impala.service.HistoricalStats.writeStats(HistoricalStats.java:126)
at org.apache.impala.service.JniFrontend.storeExecStats(JniFrontend.java:1002)
I20260805 20:38:00.835767 135178 status.cc:123] IllegalStateException: null
@ 0x137d14d impala::Status::Status()
@ 0x1d56081 impala::JniUtil::GetJniExceptionMsg()
@ 0x12c87d9 impala::JniCall::Call<>()
@ 0x198fd5d impala::Frontend::StoreExecStats()
@ 0x1a7427d impala::ImpalaServer::StoreExecutionStats()
@ 0x1a7461e impala::ImpalaServer::CloseClientRequestState()
@ 0x1a758d7 impala::ImpalaServer::FinishUnregisterQuery()
@ 0x1a94e27 boost::detail::function::void_function_obj_invoker2<>::invoke()
@ 0x1ab09e6 impala::ThreadPool<>::WorkerThread()
@ 0x1a94ed1 boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0x1df61a4 boost::function0<>::operator()()
@ 0x1df5080 impala::Thread::SuperviseThread()
@ 0x1df5ae1 boost::detail::thread_data<>::run()
@ 0x296e827 thread_proxy
@ 0x70db8ee94ac3 (unknown)
@ 0x70db8ef268c0 (unknown)
@ (nil) (unknown)
Maybe I should change the log level and test on not seeing such logs. Do you have better ideas about adding the test?
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Aug 2026 16:17:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 12:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24556/11/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/11/fe/src/main/java/org/apache/impala/planner/PlanNode.java@50
PS11, Line 50: import org.apache.impala.common.PrintUtils;
> unused import: org.apache.impala.common.Pair
Done
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 12
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Aug 2026 15:38:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23639/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 15
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Aug 2026 13:56:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 15:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24556/13/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/13/fe/src/main/java/org/apache/impala/planner/UnionNode.java@381
PS13, Line 381: sb.append(childKey);
> constExprLists_ isn't guaranteed to contain only LiteralExprs, so this cast
Nice catch! Updated this to return null on nondeterministic expressions. For constant expressions that are not LiteralExpr, e.g. "1+1", use their sql strings in the key. Note that evaluating their values might impact performance. We can revisit this in the future.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 15
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Aug 2026 13:33:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@383
PS8, Line 383: return sb.toString();
Should we include `limit_` in this key, as `HdfsScanNode` does? A top-level
`UNION ALL ... LIMIT 1` keeps the limit on the UnionNode, and the runtime
`num_rows` stored in HBO is 1. Since the current key is identical to the
unlimited union, a later unlimited query can reuse cardinality 1.
`capCardinalityAtLimit()` only protects the opposite direction. Could we add
the UnionNode limit to the key and cover limited-to-unlimited reuse in a test?
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@391
PS8, Line 391: List<PlanNode.HboKeyedNode> sortedChildren = PlanNode.buildSortedHboNodes(
I don't think sorting this list by `EXPR_REWRITE` keeps it aligned with every
strategy-specific operand list. For example:
`Q1: (year=2009 AND int_col=1) UNION ALL (year=2019 AND int_col=-999)`
`Q2: (year=2009 AND int_col=-999) UNION ALL (year=2019 AND int_col=1)`
The `IGNORE_PARTITION_CONSTANTS` Union keys are equal. However, the scan stats
for both runs are ordered by year, so positional matching compares 2009 with
2009 and 2019 with 2019. The corresponding `int_col` operands actually use
different partitions and should fail the input-row check. This can produce a
false HBO hit. Could we associate scan stats with the strategy-specific child
identity, rather than using one positional list for all strategies?
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@415
PS8, Line 415: populateHboThriftFields(msg, serialCtx);
`populateHboThriftFields()` also runs for a constants-only Union. Its key is
non-null (`constOps:N|OPERANDS:[]`), but `appendScanInputStats()` leaves the
scan list empty. `tryUpdateCardinalityFromHbo()` always skips such a run; the
first execution still stores it, and the second execution with the same key
reaches `getSimilarRunIndex()` and fails its non-empty scan-stats precondition.
Could we either skip HBO fields when the subtree has no scans or explicitly
support key-only runs? A test that executes `select 1 union all select 2` twice
with HBO storage enabled should cover this.
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/service/HistoricalStats.java
File fe/src/main/java/org/apache/impala/service/HistoricalStats.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/service/HistoricalStats.java@118
PS8, Line 118: for (int i = 0; i < runs.size(); i++) {
Similarity ranges can overlap, so returning the first match does not always
return the newest run. With a 10% threshold, write input rows 100, then 120,
then 110. The first two are retained; writing 110 removes 100 and leaves
`[120, 110]`. A lookup for 110 then matches the old 120 run before the exact
new 110 run. Could we search from newest to oldest, or remove all matching runs
on write? It would also be useful to extend `testWriteReadDedupBySize()` with
this three-run case.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sun, 02 Aug 2026 12:13:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@376
PS8, Line 376: sb.append("constOps:").append(constExprLists_.size()).append("|");
> That's a good point! I think conjuncts might be safe since they will be mig
Sounds good about putting in a future patch.
But if we're punting this, is there a need to even keep track of the number of constOps? Can we just punt this part as well to keep the string simpler?
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 18:10:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@376
PS8, Line 376: sb.append("constOps:").append(constExprLists_.size()).append("|");
> Sounds good about putting in a future patch.
Without this, the key string will be too simple for constant VALUES/SELECT clauses which don't have any children. E.g. a VALUES clause with 34 rows is represent as
CARDINALITY:UnionNode:constOps:34|OPERANDS:[]
Adding this can at least distinguish their cardinalities, then ancestor nodes (e.g. join/agg) also have different keys. So it still helps to avoid mapping PlanNodes with different cardinalities to the same key.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sat, 01 Aug 2026 01:37:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23538/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Aug 2026 13:08:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 8:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@415
PS8, Line 415: populateHboThriftFields(msg, serialCtx);
> `populateHboThriftFields()` also runs for a constants-only Union. Its key i
Thanks, the PS10 guard looks good. Could you also add the regression test from the original comment that executes a constants-only UNION twice with HBO storage enabled? That would cover the failing second-run path.
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/service/HistoricalStats.java
File fe/src/main/java/org/apache/impala/service/HistoricalStats.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/service/HistoricalStats.java@118
PS8, Line 118: for (int i = 0; i < runs.size(); i++) {
> Similarity ranges can overlap, so returning the first match does not always
Agreed that keeping 120 is useful. I wasn't suggesting that it must be removed. The issue is that it shadows the exact 110 run. With [120, 110], a lookup for 110 returns 120 before reaching 110. Searching newest-to-oldest, or otherwise preferring the closest match, would keep 120 available for a lookup of 130 while returning 110 for 110. Could we cover this with the three-run test?
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Aug 2026 14:40:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 15: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 15
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Aug 2026 19:07:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 15: Code-Review+2
(1 comment)
This is lgtm; just a nit in the commit msg. I like the improved test coverage based on the reviews. Carrying forward Aleksander and Steve's +1 and re-adding my vote.
http://gerrit.cloudera.org:8080/#/c/24556/15//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24556/15//COMMIT_MSG@43
PS15, Line 43: JoinNode
nit: you mean UnionNode
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 15
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 19 Aug 2026 04:02:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 8:
(3 comments)
Moved the missing changes of HistoricalStats from the JoinNode patch and added FE tests for matching historical runs.
http://gerrit.cloudera.org:8080/#/c/24556/7/fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
File fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java:
http://gerrit.cloudera.org:8080/#/c/24556/7/fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java@26
PS7, Line 26: import java.util.List;
> unused import: java.util.HashMap
Done
http://gerrit.cloudera.org:8080/#/c/24556/7/fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java@28
PS7, Line 28: import org.apache.impala.planner.CanonicalizationStrategy;
> unused import: java.util.Map
Done
http://gerrit.cloudera.org:8080/#/c/24556/7/fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java@31
PS7, Line 31: import org.apache.impala.thrift.TPlanNodeRunWithKeys;
> unused import: org.apache.impala.thrift.TCanonicalizationStrategy
Done
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jul 2026 10:25:45 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23458/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jul 2026 10:46:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 17:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13870/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 17
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 20 Aug 2026 01:20:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23404/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 28 Jul 2026 03:00:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 11:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@376
PS8, Line 376: if (limit_ > 0) sb.append("LIMIT:").append(limit_).append("|");
> I can't say for sure if it's better than nothing because if the cardinality
Ack
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@391
PS8, Line 391: public void appendScanInputStats(TPlanNodeRun execStats) {
> This approach sounds reasonable if the fallback order is used for both the
Done. Added an e2e test.
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@415
PS8, Line 415: populateHboThriftFields(msg, serialCtx);
> Changing the log level would only hide the failed write. Since PS10 fixes t
Good points! These verify the if-branches that we skip HBO. Added the FE test and e2e test.
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/service/HistoricalStats.java
File fe/src/main/java/org/apache/impala/service/HistoricalStats.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/service/HistoricalStats.java@118
PS8, Line 118: for (int i = 0; i < runs.size(); i++) {
> Agreed that keeping 120 is useful. I wasn't suggesting that it must be remo
Searching newest-to-oldest still have problems. E.g. when the list has only 110 and then 122 (>110*1.1) comes, the list become [110, 122]. Then 122 is newer and it's able to serve 110 since it covers the range of [122 * 0,9, 122 * 1.1] = [109, 134].
Preferring the closest match is hard to define since each run has multiple scans and the scan matching could compare numRows, catalogVersion and totalFileSize.
I think the current approach is simple but it works after enough runs given that
- The similar one will be removed in the write path and the latest one will be added to the list.
- The list has a fixed capacity so old/stale runs won't pile up.
How about leaving this for future improvement when we find it worths the effort?
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 11
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Aug 2026 14:59:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 4: Code-Review+2
LGTM. A related thing is about other set operations such as EXCEPT, INTERSECT. Would those appear in a different patch ? I tend to think of all set operations as a unit. Either way is ok.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 22 Jul 2026 23:30:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23584/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 12
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 12 Aug 2026 15:59:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@376
PS8, Line 376: sb.append("constOps:").append(constExprLists_.size()).append("|");
> So we're just counting the constant operations and not showing what's in th
That's a good point! I think conjuncts might be safe since they will be migrated into the UnionStmt:
https://github.com/apache/impala/blob/c7dd8b5f95e262065a682e5cbf80645f3bc2809a/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java#L2353-L2355
Const operands that don't survive in the predicates will be marked as hasEmptyResultSet=true then be skipped:
https://github.com/apache/impala/blob/c7dd8b5f95e262065a682e5cbf80645f3bc2809a/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java#L2297-L2308
Then at the point of creating UnionNode, the const operands are all valid.
However, if downstream PlanNodes aggregate or join on the const operand results, the cardinality might change. I think the ancestor nodes can pass down this info when generating HBO key strings of their children. Such use case seems rare so let's fix it in a future patch.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 31 Jul 2026 08:52:24 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24556
to look at the new patch set (#8).
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
HBO Key String
--------------
The HBO key string of a UnionNode consits of the number of const
operands and the key strings of its children. The children strings are
sorted so the operand order doesn't matter.
Historical Runs Matching
------------------------
If the HBO key strings match, we pick one from the list of historical
runs corresponding to that key.
UnionNode is the first HBO-supported node type that has multiple
children so we need to extend the existing matching mechanism to
supports matching multiple scan nodes.
Each UnionNode collects all the leaf scan input stats inside its subtree.
These scan input stats are sorted using the same order of the sorted
operands.
While matching a historical run, element-wise comparison of all
TScanInputStats entries is performed to ensure they are similar
respectively. Take the following JoinNode runs as an example.
UnionNode UnionNode
/ | \ / | \
A B C A' B' C'
The matching requires similar(A, A') && similar(B, B') && similar(C, C').
Similarity of a scan node pair is checked in the following way:
- If both sides have valid input_rows (come from HMS numRows stats),
compare row count.
- If either input_rows is missing,
- For EXPR_REWRITE, an exact catalog version match means similar.
if catalog versions differ, compare input file sizes.
- For other strategies, catalog version is not a reliable signal,
so compare input file sizes only.
Note that the hash key matching already ensures table names and
conjuncts are matched. It's done before finding the historical run so we
don't need to compare table names or conjuncts here.
Testing
- Added FE tests for HBO key strings.
- Added FE tests to matching historical runs.
- Added e2e tests on all UNION types.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
8 files changed, 518 insertions(+), 86 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/8
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 7:
(3 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24556/7/fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
File fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java:
http://gerrit.cloudera.org:8080/#/c/24556/7/fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java@26
PS7, Line 26: import java.util.HashMap;
unused import: java.util.HashMap
http://gerrit.cloudera.org:8080/#/c/24556/7/fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java@28
PS7, Line 28: import java.util.Map;
unused import: java.util.Map
http://gerrit.cloudera.org:8080/#/c/24556/7/fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java@31
PS7, Line 31: import org.apache.impala.thrift.TCanonicalizationStrategy;
unused import: org.apache.impala.thrift.TCanonicalizationStrategy
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 7
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 30 Jul 2026 10:20:26 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/24556/1/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/1/fe/src/main/java/org/apache/impala/planner/UnionNode.java@21
PS1, Line 21: import java.util.List;
> unused import: java.util.Collections
Done
http://gerrit.cloudera.org:8080/#/c/24556/1/fe/src/main/java/org/apache/impala/planner/UnionNode.java@370
PS1, Line 370: statsType, strategy, children_);
> line too long (110 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/24556/1/fe/src/main/java/org/apache/impala/planner/UnionNode.java@376
PS1, Line 376: constOps:"
changed this to "constOps" for const operands.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 01:30:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Aleksandr Efimov, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24556
to look at the new patch set (#15).
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
IMPALA-15127: Support HBO for UnionNode cardinality
This extends HBO to support tracking and using cardinality of
UnionNodes. Only FE changes are needed.
HBO Key String
--------------
The HBO key string of a UnionNode consits of the optional LIMIT, the
canonicalized const operands, and the key strings of its children.
To make the HBO key string consistent for different operand orders,
the children are sorted by their concatenated and sorted scan table
names. E.g. an operand with only one table scan in its subtree uses the
table name as the sort key, an operand with multiple table scans uses a
sort key like "tbl1,tbl2,tbl3".
In case when two operands have the same set of table scans, we use the
original order in query for simplicity. Another query that flips their
order will have a different HBO key, which misses the HBO stats. Such
cases are rare in practice, so we keep them unoptimized.
Historical Runs Matching
------------------------
If the HBO key strings match, we pick one from the list of historical
runs corresponding to that key.
UnionNode is the first HBO-supported node type that has multiple
children so we need to extend the existing matching mechanism to
supports matching multiple scan nodes.
Each UnionNode collects all the leaf scan input stats inside its subtree
in pre-order traversal. The order of visiting its children is the same
as the operand order used in the HBO key string.
While matching a historical run, element-wise comparison of all
TScanInputStats entries is performed to ensure they are similar
respectively. Take the following JoinNode runs as an example. The
children (sorted) are ScanNodes A, B, C.
UnionNode UnionNode
/ | \ / | \
A B C A' B' C'
The matching requires similar(A, A') && similar(B, B') && similar(C, C').
Similarity of a scan node pair is checked in the following way:
- If both sides have valid input_rows (come from HMS numRows stats),
compare row count.
- If either input_rows is missing,
- For EXPR_REWRITE, an exact catalog version match means similar.
if catalog versions differ, compare input file sizes.
- For other strategies, catalog version is not a reliable signal,
so compare input file sizes only.
Note that the hash key matching already ensures table names and
conjuncts are matched. It's done before finding the historical run so we
don't need to compare table names or conjuncts here.
For UnionNodes that have only constant operands, there are no scan input
stats. The corresponding HBO value list will have at most one item since
the HBO key matching already ensures the constants are identical.
Testing
- Added FE tests for HBO key strings.
- Added FE tests to matching historical runs.
- Added e2e tests on all UNION types.
Assisted-by: Opus 4.8 (Claude Code)
Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
---
M fe/src/main/java/org/apache/impala/planner/AggregationNode.java
M fe/src/main/java/org/apache/impala/planner/CanonicalizationStrategy.java
M fe/src/main/java/org/apache/impala/planner/ExprCanonicalizer.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/planner/ScanNode.java
M fe/src/main/java/org/apache/impala/planner/UnionNode.java
M fe/src/main/java/org/apache/impala/service/HistoricalStats.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
A fe/src/test/java/org/apache/impala/service/HistoricalStatsTest.java
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-collection-scan.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-consts.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union-same-tables.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-union.test
M tests/query_test/test_hbo.py
14 files changed, 854 insertions(+), 95 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/24556/15
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 15
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 13:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@376
PS8, Line 376: if (!constExprLists_.isEmpty()) {
> Ack
PS13 addes the constants into the key.
http://gerrit.cloudera.org:8080/#/c/24556/8/fe/src/main/java/org/apache/impala/planner/UnionNode.java@415
PS8, Line 415: }
> PS12 still leaves a no-scan path through a parent node. `select 1 union sel
Extended the code to allow empty scan input stats for matching const-only UnionNodes. Also added the constants into the HBO key string so the matching ensures constants are identical. Note that we don't want to track const-only UnionNodes since their cardinalities are clear and accurate. However, for ancestor nodes of const-only UnionNodes, planner can only estimate the cardinality so we track them in HBO.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 13
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 14 Aug 2026 05:48:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 17: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 17
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 20 Aug 2026 06:22:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15127: Support HBO for UnionNode cardinality
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24556 )
Change subject: IMPALA-15127: Support HBO for UnionNode cardinality
......................................................................
Patch Set 13:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24556/13/fe/src/main/java/org/apache/impala/planner/UnionNode.java
File fe/src/main/java/org/apache/impala/planner/UnionNode.java:
http://gerrit.cloudera.org:8080/#/c/24556/13/fe/src/main/java/org/apache/impala/planner/UnionNode.java@381
PS13, Line 381: .map(c -> ExprCanonicalizer.canonicalizeLiteral((LiteralExpr) c))
> constExprLists_ isn't guaranteed to contain only LiteralExprs, so this cast
This looks good to me now. Thanks for covering both cases in the tests.
--
To view, visit http://gerrit.cloudera.org:8080/24556
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie228f530bdcb171d3b717966673164bf9a4c45c8
Gerrit-Change-Number: 24556
Gerrit-PatchSet: 13
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 17 Aug 2026 19:01:29 +0000
Gerrit-HasComments: Yes