You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org> on 2026/04/08 13:18:46 UTC
[Impala-ASF-CR] IMPALA-14823: Lazy load IcebergContentFileStore in Local Catalog mode
Noemi Pap-Takacs has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24091
Change subject: IMPALA-14823: Lazy load IcebergContentFileStore in Local Catalog mode
......................................................................
IMPALA-14823: Lazy load IcebergContentFileStore in Local Catalog mode
This patch optimizes file metadata loading for Iceberg tables in the
Local Catalog mode. Previously, loading an Iceberg table would eagerly
fetch the entire list of content files from CatalogD. Serializing and
sending `IcebergContentFileStore` caused significant overhead for
metadata-only operations like DESCRIBE or SHOW TABLE STATS, which do
not require file-level details.
The `IcebergContentFileStore`, which holds the file list, is now
lazily loaded.
- The `TGetPartialCatalogObjectRequest` thrift structure and
`IcebergTable.getPartialInfo` in CatalogD were updated to support
requesting Iceberg metadata without content files (`DESCRIPTOR_ONLY`).
- `LocalIcebergTable` now initializes with minimal metadata. Accessing
`getContentFileStore()` (required for query planning) triggers a
subsequent RPC to fetch the file list.
- `FeIcebergTable.Utils.getTableStats` was optimized to use the
Iceberg snapshot summary for file counts instead of counting loaded
files, ensuring `SHOW TABLE STATS` remains lightweight.
Testing:
- Added `LocalIcebergContentFileStoreTest` unit test to verify that
file loading is deferred until explicitly accessed.
Generated by Gemini 3 Pro
Change-Id: If305129a0f5a6e6fa102dfb5628c750ab5318e03
---
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
A fe/src/test/java/org/apache/impala/catalog/local/LocalIcebergContentFileStoreTest.java
M fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
12 files changed, 166 insertions(+), 24 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/24091/2
--
To view, visit http://gerrit.cloudera.org:8080/24091
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If305129a0f5a6e6fa102dfb5628c750ab5318e03
Gerrit-Change-Number: 24091
Gerrit-PatchSet: 2
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14823: Lazy load IcebergContentFileStore in Local Catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24091 )
Change subject: IMPALA-14823: Lazy load IcebergContentFileStore in Local Catalog mode
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22093/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24091
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If305129a0f5a6e6fa102dfb5628c750ab5318e03
Gerrit-Change-Number: 24091
Gerrit-PatchSet: 2
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Apr 2026 13:41:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14823: Lazy load IcebergContentFileStore in Local Catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24091 )
Change subject: IMPALA-14823: Lazy load IcebergContentFileStore in Local Catalog mode
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13303/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24091
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If305129a0f5a6e6fa102dfb5628c750ab5318e03
Gerrit-Change-Number: 24091
Gerrit-PatchSet: 2
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Apr 2026 13:19:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14823: Lazy load IcebergContentFileStore in Local Catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24091 )
Change subject: IMPALA-14823: Lazy load IcebergContentFileStore in Local Catalog mode
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13303/
--
To view, visit http://gerrit.cloudera.org:8080/24091
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If305129a0f5a6e6fa102dfb5628c750ab5318e03
Gerrit-Change-Number: 24091
Gerrit-PatchSet: 2
Gerrit-Owner: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Apr 2026 17:41:08 +0000
Gerrit-HasComments: No