You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org> on 2026/09/22 11:42:56 UTC
[Impala-ASF-CR] IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
Aleksandr Efimov has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24913
Change subject: IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
......................................................................
IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
An HdfsScanNode with the count(*) optimization reads row counts from
file metadata and returns one row per file or row group. Its cardinality
is the number of files, and HBO stores that under the same key as a
regular scan of the same table and conjuncts, because
generateHboKeyString() does not include countStarSlot_. The next regular
scan then takes the file count as its cardinality from HBO.
For example, after "SELECT count(*) FROM t" on a Parquet table with 15
files and 13.66M rows, "SELECT * FROM t" is planned with
cardinality=15 (from HBO), and every estimate above the scan follows.
This patch adds a <COUNT_STAR> marker to the scan key when the
optimization is applied, so the two kinds of runs get separate keys.
Testing:
- Added HboKeyStringTest.testCountStarScanNodeKey.
Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
---
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
2 files changed, 28 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/24913/1
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 1
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
[Impala-ASF-CR] IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24913
to look at the new patch set (#2).
Change subject: IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
......................................................................
IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
An HdfsScanNode with the count(*) optimization reads row counts from
file metadata and returns one row per file or row group. Its cardinality
is the number of files, and HBO stores that under the same key as a
regular scan of the same table and conjuncts, because
generateHboKeyString() does not include countStarSlot_. The next regular
scan then takes the file count as its cardinality from HBO.
For example, after "SELECT count(*) FROM t" on a Parquet table with 15
files and 13.66M rows, "SELECT * FROM t" is planned with
cardinality=15 (from HBO), and every estimate above the scan follows.
This patch adds a <COUNT_STAR> marker to the scan key when the
optimization is applied, so the two kinds of runs get separate keys.
Testing:
- Added HboKeyStringTest.testCountStarScanNodeKey.
Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Assisted-by: claude-opus-5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
2 files changed, 28 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/24913/2
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24913/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24913/3//COMMIT_MSG@20
PS3, Line 20: input rows count whole partitions (or all files of an Iceberg table).
> Yeah, we can do it in a follow-up JIRA.
Filed IMPALA-15458.
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Sep 2026 09:43:43 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14135/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 23 Sep 2026 20:43:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24913
to look at the new patch set (#4).
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
IMPALA-15394: Add missing scan properties to the HBO scan key
HdfsScanNode.generateHboKeyString() keys a scan by its table and
conjuncts. Some scans of the same table and conjuncts return different
rows, and their input stats match too, so HBO gives one the cardinality
of the other:
- An optimized count(*) scan returns one row per file or row group.
After "SELECT count(*) FROM t" on a Parquet table with 15 files and
13.66M rows, "SELECT * FROM t" is planned with cardinality=15 (from
HBO), and every estimate above the scan follows.
- A partition key scan, e.g. "SELECT DISTINCT year FROM t", returns one
row per scan range.
- A sampled scan (TABLESAMPLE) reads only part of the files, while its
input rows count every partition with a sampled file (or all files of
an Iceberg table). Without numRows, runs match by catalog version.
- Conjuncts on the items of a collection, together with the
IsNotEmptyPredicate of the join, filter the rows of the scan. On
functional_parquet.complextypestbl the scan returned 2 rows for
"a.item > 1" and 0 for "a.item > 100" under the same key.
This patch adds <COUNT_STAR>, <PARTITION_KEY_SCAN> and <SAMPLE:percent>
markers and the canonicalized collection conjuncts to the scan key. The
collection is named by its canonical path, so the key doesn't depend on
the aliases. The seed of a sample is left out: it picks other files but
reads the same share of the table.
Testing:
- Added HboKeyStringTest.testCountStarScanNodeKey,
testPartitionKeyAndSampledScanNodeKeys and
testCollectionConjunctsInScanNodeKey. Removing each marker or the
collection conjuncts, or naming the collection by its alias, fails
the test that covers it.
- Added e2e cases to test_hbo.py: a regular scan after a count(*) scan,
and item conjuncts of a joined collection. Without this patch both
get the cardinality of the other query from HBO.
Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Assisted-by: claude-opus-5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-collection-scan.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-count-star-scan.test
M tests/query_test/test_hbo.py
5 files changed, 199 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/24913/4
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 4
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
IMPALA-15394: Add missing scan properties to the HBO scan key
HdfsScanNode.generateHboKeyString() keys a scan by its table and
conjuncts. Some scans of the same table and conjuncts return different
rows, and their input stats match too, so HBO gives one the cardinality
of the other:
- An optimized count(*) scan returns one row per file or row group.
After "SELECT count(*) FROM t" on a Parquet table with 15 files and
13.66M rows, "SELECT * FROM t" is planned with cardinality=15 (from
HBO), and every estimate above the scan follows.
- A partition key scan, e.g. "SELECT DISTINCT year FROM t", returns one
row per scan range.
- A sampled scan (TABLESAMPLE) reads only part of the files, while its
input rows count every partition with a sampled file (or all files of
an Iceberg table).
- Conjuncts on the items of a collection, together with the
IsNotEmptyPredicate of the join, filter the rows of the scan. On
functional_parquet.complextypestbl the scan returned 2 rows for
"a.item > 1" and 0 for "a.item > 100" under the same key.
This patch adds <COUNT_STAR>, <PARTITION_KEY_SCAN> and <SAMPLE:percent>
markers and the canonicalized collection conjuncts to the scan key. The
collection is named by its canonical path, so the key doesn't depend on
the aliases. The seed of a sample is left out: it picks other files but
reads the same share of the table.
Testing:
- Added HboKeyStringTest.testCountStarScanNodeKey,
testPartitionKeyAndSampledScanNodeKeys and
testCollectionConjunctsInScanNodeKey. Removing each marker or the
collection conjuncts, or naming the collection by its alias, fails
the test that covers it.
- Added e2e cases to test_hbo.py: a regular scan after a count(*) scan
or a partition key scan, and item conjuncts of a joined collection.
Without this patch each gets the cardinality of the other query from
HBO.
Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Assisted-by: claude-opus-5 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24913
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-collection-scan.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-count-star-scan.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-partition-key-scan.test
M tests/query_test/test_hbo.py
6 files changed, 234 insertions(+), 0 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 8
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 5:
(3 comments)
Added test_partition_key_scan_cardinality in PS6. On a cluster without this patch, the regular scan after a partition key scan gets cardinality=4 (from HBO) instead of 1.22K.
http://gerrit.cloudera.org:8080/#/c/24913/5/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/24913/5/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1941
PS5, Line 1941: // key scan one row per scan range, not the table's rows, while their input stats
> nit: add "returns"
Done
http://gerrit.cloudera.org:8080/#/c/24913/5/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1943
PS5, Line 1943: // the files, and without numRows its runs are matched by the catalog version alone.
> nit: remove this? It's a bit confusing. We also match total file bytes and
Removed it here and in the commit message.
http://gerrit.cloudera.org:8080/#/c/24913/5/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24913/5/tests/query_test/test_hbo.py@247
PS5, Line 247: # expected cardinality doesn't depend on the size of the data files.
> nit: "Use a parquet table with stats."
Done
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Sep 2026 09:43:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24913
to look at the new patch set (#5).
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
IMPALA-15394: Add missing scan properties to the HBO scan key
HdfsScanNode.generateHboKeyString() keys a scan by its table and
conjuncts. Some scans of the same table and conjuncts return different
rows, and their input stats match too, so HBO gives one the cardinality
of the other:
- An optimized count(*) scan returns one row per file or row group.
After "SELECT count(*) FROM t" on a Parquet table with 15 files and
13.66M rows, "SELECT * FROM t" is planned with cardinality=15 (from
HBO), and every estimate above the scan follows.
- A partition key scan, e.g. "SELECT DISTINCT year FROM t", returns one
row per scan range.
- A sampled scan (TABLESAMPLE) reads only part of the files, while its
input rows count every partition with a sampled file (or all files of
an Iceberg table). Without numRows, runs match by catalog version.
- Conjuncts on the items of a collection, together with the
IsNotEmptyPredicate of the join, filter the rows of the scan. On
functional_parquet.complextypestbl the scan returned 2 rows for
"a.item > 1" and 0 for "a.item > 100" under the same key.
This patch adds <COUNT_STAR>, <PARTITION_KEY_SCAN> and <SAMPLE:percent>
markers and the canonicalized collection conjuncts to the scan key. The
collection is named by its canonical path, so the key doesn't depend on
the aliases. The seed of a sample is left out: it picks other files but
reads the same share of the table.
Testing:
- Added HboKeyStringTest.testCountStarScanNodeKey,
testPartitionKeyAndSampledScanNodeKeys and
testCollectionConjunctsInScanNodeKey. Removing each marker or the
collection conjuncts, or naming the collection by its alias, fails
the test that covers it.
- Added e2e cases to test_hbo.py: a regular scan after a count(*) scan,
and item conjuncts of a joined collection. Without this patch both
get the cardinality of the other query from HBO.
Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Assisted-by: claude-opus-5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-collection-scan.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-count-star-scan.test
M tests/query_test/test_hbo.py
5 files changed, 191 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/24913/5
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24258/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 6
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Sep 2026 10:08:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 6: Code-Review+2
LGTM. Thanks for fixing these!
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 6
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 02 Oct 2026 12:11:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24913/3/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24913/3/tests/query_test/test_hbo.py@195
PS3, Line 195:
> Can we improve this e2e test to cover the bug of missing collection conjunc
Covered in PS6: the test stores history for `a.item > 1` and checks that `a.item > 100` does not reuse it. All 32 checks pass; removing collection predicates from the key makes the last check fail.
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 6
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 02 Oct 2026 08:51:17 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24913
to look at the new patch set (#3).
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
IMPALA-15394: Add missing scan properties to the HBO scan key
HdfsScanNode.generateHboKeyString() keys a scan by its table and
conjuncts. Some scans of the same table and conjuncts return different
rows, and their input stats match too, so HBO gives one the cardinality
of the other:
- An optimized count(*) scan returns one row per file or row group.
After "SELECT count(*) FROM t" on a Parquet table with 15 files and
13.66M rows, "SELECT * FROM t" is planned with cardinality=15 (from
HBO), and every estimate above the scan follows.
- A partition key scan, e.g. "SELECT DISTINCT year FROM t", returns one
row per scan range.
- A sampled scan (TABLESAMPLE) reads only part of the files, while its
input rows count whole partitions (or all files of an Iceberg table).
- Conjuncts on the items of a collection, together with the
IsNotEmptyPredicate of the join, filter the rows of the scan. On
functional_parquet.complextypestbl the scan returned 2 rows for
"a.item > 1" and 0 for "a.item > 100" under the same key.
This patch adds <COUNT_STAR>, <PARTITION_KEY_SCAN> and <SAMPLE:percent>
markers and the canonicalized collection conjuncts to the scan key. The
collection is named by its canonical path, so the key doesn't depend on
the aliases. The seed of a sample is left out: it picks other files but
reads the same share of the table.
Testing:
- Added HboKeyStringTest.testCountStarScanNodeKey,
testPartitionKeyAndSampledScanNodeKeys and
testCollectionConjunctsInScanNodeKey. Removing each marker or the
collection conjuncts, or naming the collection by its alias, fails
the test that covers it.
Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Assisted-by: claude-opus-5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
2 files changed, 97 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/24913/3
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 3:
(7 comments)
It'd be nice to add some e2e tests.
http://gerrit.cloudera.org:8080/#/c/24913/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24913/3//COMMIT_MSG@20
PS3, Line 20: input rows count whole partitions (or all files of an Iceberg table).
We use sampledPartitions_ in this case by getSampledOrRawPartitions(). But it's still incorrect if not all files in the sampledPartitions are selected. Maybe we can use the sample percent to fix this.
I think the cardinality of a sampled scan can be used to estimate the corresponding full scan. Or if the full scan with more partition predicates have the same input cardinality of the sampled scan, they can share the cardinality. Then we don't need the sample clause in HBO key string.
What do you think?
http://gerrit.cloudera.org:8080/#/c/24913/3//COMMIT_MSG@24
PS3, Line 24: "a.item > 1" and 0 for "a.item > 100" under the same key.
Nice catch!
http://gerrit.cloudera.org:8080/#/c/24913/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/24913/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1943
PS3, Line 1943: which has the same input stats.
nit: sampled scan might not has the same input stats as the full scan's.
http://gerrit.cloudera.org:8080/#/c/24913/3/fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
File fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java:
http://gerrit.cloudera.org:8080/#/c/24913/3/fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java@149
PS3, Line 149: assertTrue(countStarScan.getCountStarSlot() != null);
nit: I think we don't need to verify this since it's unrelevant to HBO. Then we can use scanNodeKey() as well in this test.
http://gerrit.cloudera.org:8080/#/c/24913/3/fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java@164
PS3, Line 164: scanNodeKey
nit: maybe use a more specific name like singleScanKey, uniqueScanKey, etc.
http://gerrit.cloudera.org:8080/#/c/24913/3/fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java@213
PS3, Line 213: "functional_parquet.complextypestbl t, t.int_array x WHERE x.item > 1"));
Can we also assert the actual key strings? HboKeyStringTest is a place to show how the HBO key strings look like.
http://gerrit.cloudera.org:8080/#/c/24913/3/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24913/3/tests/query_test/test_hbo.py@195
PS3, Line 195: def test_collection_scan_cardinality(self):
Can we improve this e2e test to cover the bug of missing collection conjuncts?
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Sep 2026 10:50:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 3:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/24913/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24913/3//COMMIT_MSG@24
PS3, Line 24: "a.item > 1" and 0 for "a.item > 100" under the same key.
> Nice catch!
Thanks!
http://gerrit.cloudera.org:8080/#/c/24913/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/24913/3/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1943
PS3, Line 1943: // same table, which has the same input stats.
> nit: sampled scan might not has the same input stats as the full scan's.
Done
http://gerrit.cloudera.org:8080/#/c/24913/3/fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
File fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java:
http://gerrit.cloudera.org:8080/#/c/24913/3/fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java@149
PS3, Line 149: assertTrue(countStarScan.getCountStarSlot() != null);
> nit: I think we don't need to verify this since it's unrelevant to HBO. The
Done
http://gerrit.cloudera.org:8080/#/c/24913/3/fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java@164
PS3, Line 164: private String scanNodeKey(String query) throws ImpalaException {
> nit: maybe use a more specific name like singleScanKey, uniqueScanKey, etc.
Done, singleScanKey()
http://gerrit.cloudera.org:8080/#/c/24913/3/fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java@213
PS3, Line 213: "functional_parquet.complextypestbl t, t.int_array x WHERE x.item > 1"));
> Can we also assert the actual key strings? HboKeyStringTest is a place to s
Done
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Sep 2026 17:50:43 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24177/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 25 Sep 2026 06:25:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24913/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24913/3//COMMIT_MSG@20
PS3, Line 20: input rows count whole partitions (or all files of an Iceberg table).
> We use sampledPartitions_ in this case by getSampledOrRawPartitions(). But
Agreed, input_rows is too high there. But I'd keep <SAMPLE> in the key: without numRows only the EXPR_REWRITE key is stored and runs match by catalog version, so a sampled and a full scan of such a table would share the cardinality. Fixing input_rows for sampled scans could be a follow-up JIRA. WDYT?
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Sep 2026 17:38:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14189/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 02 Oct 2026 12:12:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 7: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 02 Oct 2026 16:41:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 5:
PS5: the count(*) test now uses tpch_parquet.orders. It has stats, so the expected cardinality no longer depends on file sizes (503 vs 504 in the dryrun).
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 25 Sep 2026 06:07:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Sep 2026 01:10:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24913/2/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/24913/2/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1942
PS2, Line 1942: if (countStarSlot_ != null) sb.append("|<COUNT_STAR>");
> This reminds me there are some fields that are missing in the HBO key strin
Thanks, both were missing: they shared the key and input stats of a regular scan. PS3 adds <PARTITION_KEY_SCAN> and <SAMPLE:percent>. It also adds conjuncts on collection items, which filter the scan rows through the IsNotEmptyPredicate and were missing too.
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 23 Sep 2026 10:13:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14153/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 25 Sep 2026 11:14:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 25 Sep 2026 15:40:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 5:
(4 comments)
Thanks for adding the e2e tests. Do you mind adding a test for partition scan queries as well?
Just have some nits on comments which seem too verbose.
http://gerrit.cloudera.org:8080/#/c/24913/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24913/3//COMMIT_MSG@20
PS3, Line 20: input rows count every partition with a sampled file (or all files of
> Agreed, input_rows is too high there. But I'd keep <SAMPLE> in the key: wit
Yeah, we can do it in a follow-up JIRA.
http://gerrit.cloudera.org:8080/#/c/24913/5/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/24913/5/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1941
PS5, Line 1941:
nit: add "returns"
http://gerrit.cloudera.org:8080/#/c/24913/5/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1943
PS5, Line 1943: and without numRows its runs are matched by the catalog version alone
nit: remove this? It's a bit confusing. We also match total file bytes and it's the same mechanism in matching runs of other keys.
http://gerrit.cloudera.org:8080/#/c/24913/5/tests/query_test/test_hbo.py
File tests/query_test/test_hbo.py:
http://gerrit.cloudera.org:8080/#/c/24913/5/tests/query_test/test_hbo.py@246
PS5, Line 246: The table has stats, so the
: # expected cardinality doesn't depend on the size of the data files.
nit: "Use a parquet table with stats."
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 5
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 29 Sep 2026 14:18:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24133/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 1
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 22 Sep 2026 12:03:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
......................................................................
Patch Set 2:
(1 comment)
Nice catch!
http://gerrit.cloudera.org:8080/#/c/24913/2/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java:
http://gerrit.cloudera.org:8080/#/c/24913/2/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1942
PS2, Line 1942: if (countStarSlot_ != null) sb.append("|<COUNT_STAR>");
This reminds me there are some fields that are missing in the HBO key string. Should we also add an optional string for isPartitionKeyScan_ ? Could you also check sampleParams_ ?
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 23 Sep 2026 09:32:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24913
to look at the new patch set (#6).
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
IMPALA-15394: Add missing scan properties to the HBO scan key
HdfsScanNode.generateHboKeyString() keys a scan by its table and
conjuncts. Some scans of the same table and conjuncts return different
rows, and their input stats match too, so HBO gives one the cardinality
of the other:
- An optimized count(*) scan returns one row per file or row group.
After "SELECT count(*) FROM t" on a Parquet table with 15 files and
13.66M rows, "SELECT * FROM t" is planned with cardinality=15 (from
HBO), and every estimate above the scan follows.
- A partition key scan, e.g. "SELECT DISTINCT year FROM t", returns one
row per scan range.
- A sampled scan (TABLESAMPLE) reads only part of the files, while its
input rows count every partition with a sampled file (or all files of
an Iceberg table).
- Conjuncts on the items of a collection, together with the
IsNotEmptyPredicate of the join, filter the rows of the scan. On
functional_parquet.complextypestbl the scan returned 2 rows for
"a.item > 1" and 0 for "a.item > 100" under the same key.
This patch adds <COUNT_STAR>, <PARTITION_KEY_SCAN> and <SAMPLE:percent>
markers and the canonicalized collection conjuncts to the scan key. The
collection is named by its canonical path, so the key doesn't depend on
the aliases. The seed of a sample is left out: it picks other files but
reads the same share of the table.
Testing:
- Added HboKeyStringTest.testCountStarScanNodeKey,
testPartitionKeyAndSampledScanNodeKeys and
testCollectionConjunctsInScanNodeKey. Removing each marker or the
collection conjuncts, or naming the collection by its alias, fails
the test that covers it.
- Added e2e cases to test_hbo.py: a regular scan after a count(*) scan
or a partition key scan, and item conjuncts of a joined collection.
Without this patch each gets the cardinality of the other query from
HBO.
Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Assisted-by: claude-opus-5 (Claude Code)
---
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/planner/HboKeyStringTest.java
M testdata/workloads/functional-query/queries/QueryTest/hbo-collection-scan.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-count-star-scan.test
A testdata/workloads/functional-query/queries/QueryTest/hbo-partition-key-scan.test
M tests/query_test/test_hbo.py
6 files changed, 234 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/24913/6
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 6
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/24172/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 4
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Sep 2026 18:08:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 4: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/14148/
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 4
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Sep 2026 22:31:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 3: Verified-1
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/24144/ : ABORTED
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 3
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 23 Sep 2026 13:13:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14121/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 22 Sep 2026 19:34:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Keep count(*) scans out of the HBO key of a regular scan
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 2
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 23 Sep 2026 00:04:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/14148/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 4
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Sep 2026 18:04:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15394: Add missing scan properties to the HBO scan key
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24913 )
Change subject: IMPALA-15394: Add missing scan properties to the HBO scan key
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I03183aa998f90bf834dba4ce9ad2f2e57fef1140
Gerrit-Change-Number: 24913
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 02 Oct 2026 12:12:08 +0000
Gerrit-HasComments: No