You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org> on 2026/10/01 22:54:02 UTC
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Hello Quanlong Huang, Steve Carlin,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/25002
to review the following change.
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
IMPALA-15059: Use HBO stats for Calcite filters
Use matching HDFS scan history for filters above table scans,
including column projections introduced by trimming. Reuse predicate
conversion and pruning with an isolated Analyzer. Cache lookups by
canonical key for stable estimates across optimizer alternatives.
Keep selectivity estimates when no exact history matches.
Tests: TestCalciteHboStats (23 passed). TestCalciteStats: 29 passed,
8 failures identical to the parent PS1.
Assisted-by: GPT-6 (Codex)
Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
---
M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdRowCount.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TestCalciteHboStats.java
3 files changed, 264 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/25002/1
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 1
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/25002 )
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/25002/1/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdRowCount.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdRowCount.java:
http://gerrit.cloudera.org:8080/#/c/25002/1/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdRowCount.java@75
PS1, Line 75: RelNode scanInput = RelUtil.unwrapRelNode(input);
> Can we put all (or most) of this new code in a separate method. I think it'
Sure. 25001 opens fine for me: https://gerrit.cloudera.org/c/25001/
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 1
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Oct 2026 18:32:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/25002
to look at the new patch set (#2).
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
IMPALA-15059: Use HBO stats for Calcite filters
Use matching HDFS scan history for filters above table scans,
including column projections introduced by trimming. Reuse predicate
conversion and pruning with an isolated Analyzer. Cache lookups by
canonical key for stable estimates across optimizer alternatives.
Keep selectivity estimates when no exact history matches.
Extract filtered HBO lookup into a helper to keep getRowCount small.
Tests: TestCalciteHboStats (29 passed). TestCalciteStats: 29 passed,
8 failures identical to parent 25001 PS3.
Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Assisted-by: GPT-6 (Codex)
---
M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdRowCount.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TestCalciteHboStats.java
3 files changed, 276 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/25002/2
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/25002 )
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/25002/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java:
http://gerrit.cloudera.org:8080/#/c/25002/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java@103
PS2, Line 103: private final Map<String, Double> filteredHboRowCounts_ = new HashMap<>();
nit: can we add some comments about these variables?
http://gerrit.cloudera.org:8080/#/c/25002/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java@128
PS2, Line 128: hboRowCount_ = loadHboRowCount();
Do we still need this row count if there is a Filter over TableScan and hits HBO stats? I think the Filter and TableScan (plus some corresponding Projects) will be converted into a single HdfsScanNode. Fetching HBO stats for the TableScan seems redundant and increases the planning time.
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Oct 2026 09:39:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25002 )
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Oct 2026 13:46:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25002 )
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24481/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Oct 2026 11:54:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/25002 )
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
Patch Set 1:
(1 comment)
Hmmm...I'm having trouble accessing the parent, https://gerrit.cloudera.org/#/c/25001/
Can you resubmit that one?
http://gerrit.cloudera.org:8080/#/c/25002/1/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdRowCount.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdRowCount.java:
http://gerrit.cloudera.org:8080/#/c/25002/1/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdRowCount.java@75
PS1, Line 75: RelNode scanInput = RelUtil.unwrapRelNode(input);
Can we put all (or most) of this new code in a separate method. I think it's easier to read if the top level "getRowCount" method stays small-ish and contains mostly top level-ish operations.
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 1
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Oct 2026 18:14:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25002 )
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24450/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Oct 2026 11:05:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/25002 )
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/25002/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java:
http://gerrit.cloudera.org:8080/#/c/25002/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java@103
PS2, Line 103: private final Map<String, Double> filteredHboRowCounts_ = new HashMap<>();
> nit: can we add some comments about these variables?
Added comments for both caches locally.
http://gerrit.cloudera.org:8080/#/c/25002/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java@128
PS2, Line 128: hboRowCount_ = loadHboRowCount();
> Do we still need this row count if there is a Filter over TableScan and hit
You're right, we don't need the unfiltered lookup when the Filter hits HBO. I've made it lazy locally, keeping it for fallback estimates and separate TableScan metadata requests. I still need to run the tests before uploading.
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Oct 2026 11:18:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25002 )
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14336/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Oct 2026 09:11:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Steve Carlin, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/25002
to look at the new patch set (#3).
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
IMPALA-15059: Use HBO stats for Calcite filters
Use matching HDFS scan history for filters above table scans,
including column projections introduced by trimming. Reuse predicate
conversion and pruning with an isolated Analyzer. Cache lookups by
canonical key for stable estimates across optimizer alternatives.
Keep selectivity estimates when no exact history matches.
Extract filtered HBO lookup into a helper to keep getRowCount small.
Read unfiltered history on demand and cache hits and misses so an
exact filtered match does not need the constructor lookup.
Tests: TestCalciteHboStats (30 passed). TestCalciteStats: 29 passed,
8 failures identical to 25002 PS2, including expected/actual values.
Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Assisted-by: GPT-6 (Codex)
---
M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdRowCount.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TestCalciteHboStats.java
3 files changed, 307 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/25002/3
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/25002 )
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/25002/1/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java:
http://gerrit.cloudera.org:8080/#/c/25002/1/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java@108
PS1, Line 108: private SimplifiedAnalyzer hboAnalyzer_;
> I will review this more closely (prolly in the coming week), but something
I tried a few ways to avoid the separate analyzer. Using the main analyzer added 13 slots and increased privilege requests from 2 to 14 for `select id ... where id = 1`. A non-materialized tuple avoids serialization and extra column requests, but still changes descriptors, aliases and expression counts.
Detached descriptors failed for complex pruning, e.g. `year + month = 2010`, because the pruner resolves slots through `analyzer.getDescTbl()`. An adapter with its own descriptors, expression counter and constant folder passed the focused checks, but depends more on Analyzer internals. Reusing the metadata TableRef for physical scans gave self-join operands one tuple ID and materialized 11 columns per scan instead of one. All 23 existing HBO tests still passed; additional invariant checks caught this.
A shared, isolated analyzer passed all 36 HBO tests in the join stack and matched lookup/pruning results for eight additional predicates. Focused SQL checks also returned identical results for 11 queries with HBO disabled, empty history and collected history. Fallback was disabled, and actual metadata hits were confirmed.
The planning benchmark showed no clear speed improvement. The shared prototype saved about 3% in metadata setup allocations, but allocated about 6% more during full three-table planning. It doesn't cache scan contexts across join alternatives, so this doesn't rule out a better query-scoped design.
My proposal is to keep a lazy, per-table metadata context for this patch, with final fields inside it. Laziness avoids unused setup; eager initialization passed too. Would that address your concern, or would you prefer query-scoped ownership?
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 1
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sat, 03 Oct 2026 22:34:37 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/25002 )
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/25002/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java:
http://gerrit.cloudera.org:8080/#/c/25002/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java@103
PS2, Line 103: private final Map<String, Double> filteredHboRowCounts_ = new HashMap<>();
> nit: can we add some comments about these variables?
Added comments for both caches in PS3.
http://gerrit.cloudera.org:8080/#/c/25002/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java@128
PS2, Line 128: hboRowCount_ = loadHboRowCount();
> Do we still need this row count if there is a Filter over TableScan and hit
Fixed in PS3: the unfiltered lookup is lazy and caches hits and misses. An exact filtered HBO hit skips it. Added a regression test; the full dry run passed.
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Oct 2026 15:20:05 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/25002 )
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/25002/1/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java:
http://gerrit.cloudera.org:8080/#/c/25002/1/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java@108
PS1, Line 108: private SimplifiedAnalyzer hboAnalyzer_;
I will review this more closely (prolly in the coming week), but something seems off to me here without a deep review, and I wanted to give a heads-up at what I will be looking at. It's possible that it is fine too.
I do understand there are situations where we don't use "final" variables. I tried to keep them to cache values.
That might be the case here too, but a lazy analyzer and tableref (not sure about inputexprs yet) is something I'd like to investigate why.
Also, in general, I'd like to be really sure we need a separate analyzer instance. I know I've used one in the past, but somehow I always figured out a way where I didn't need to use a separate one.
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 1
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Sat, 03 Oct 2026 19:53:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/25002 )
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24293/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 1
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 01 Oct 2026 23:16:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15059: Use HBO stats for Calcite filters
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/25002 )
Change subject: IMPALA-15059: Use HBO stats for Calcite filters
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/25002/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java:
http://gerrit.cloudera.org:8080/#/c/25002/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java@103
PS2, Line 103: private final Map<String, Double> filteredHboRowCounts_ = new HashMap<>();
> Added comments for both caches locally.
Resolved in PS3.
http://gerrit.cloudera.org:8080/#/c/25002/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java@128
PS2, Line 128: hboRowCount_ = loadHboRowCount();
> You're right, we don't need the unfiltered lookup when the Filter hits HBO.
Resolved in PS3.
--
To view, visit http://gerrit.cloudera.org:8080/25002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9270a5f94d42a4ad20d3337819b46467e83a3bf7
Gerrit-Change-Number: 25002
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Oct 2026 15:32:02 +0000
Gerrit-HasComments: Yes