You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2026/02/26 21:57:16 UTC

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24049


Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................

POC: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true, load files on
the coordintor for Iceberg tables instead of getting from
the catalog. This is inefficient at the moment as catalogd
still loads the files but not really uses them. The long
term goal is to load only minimal info for the table on
catalogd side.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex pair. This would be
suitable for REST catalog too if the key contained
snapshot ID instead of catalog version.

Pros:
- File descs are not transferred in getPartialCatalogObject RPC
- Size of cache objects seem to decrease:
  ~543->~431 MB for 1M file table
- Plans look faster due to skipping construction of
  IcebergFileContentStore (DESCRIBE 1M file table 0.5s->1ms)

Cons:
- Initial table loading is ~doubled as both the catalog and
  the coordinator need to load the files.
- No incremental loading, any catalog version change leads
  the reloaiding the files from scratch on coordinator side.

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
13 files changed, 161 insertions(+), 28 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/1
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21905/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Mar 2026 17:27:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 9:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13353/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 20 Apr 2026 19:43:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 7:

(3 comments)

answered some of the high level questions

created a review that includes some part of this change and caches IcebergContentFileStore: https://gerrit.cloudera.org/#/c/24177/

it is also in WIP state, but is much closer to being mergable than this one

http://gerrit.cloudera.org:8080/#/c/24049/7//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24049/7//COMMIT_MSG@15
PS7, Line 15: This assumes that local catalog mode is used, I didn't
            : test without it.
> To move out of PoC, will need some validation to ensure local catalog mode 
yes - ideally legacy catalog mode would be dropped, but if it is not possible, than we have to enforce that the table is loaded on catalogd


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@570
PS7, Line 570:     // TODO: does it make sense to skip file loading here?
> Where else would file loading be skipped if not here?
This patch was mainly aimed at Iceberg tables from catalog, while this file deals with the Iceberg REST catalog. I didn't think that use case through properly.


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java@219
PS7, Line 219:   // TODO: this doesn't really make sense
> It might not right now, but in the future it could.  The `tryAllProviders` 
My problem with MultiMetaProvider is that it checks each catalog in each function, not just discovery functions. E.g. if a table is discovered at a specific catalog, we should always load it from that one, and not from any catalog around. I think that this could be nicely implemented by storing the MetaProvider in TableMetaRef, but I would not deal with this in the current patch.



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 09 Apr 2026 18:37:46 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Jason Fehr, Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#11).

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................

POC IMPALA-14805: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true:
- catalogd will skip listing files while loading
  Iceberg tables
- coordinator will load the files instead and cache
  them

This assumes that local catalog mode is used, I didn't
test without it.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex + partition stats.
This would be suitable for REST catalog too if the key
contained snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak pointer to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros:
- File descs are not transferred in getPartialCatalogObject RPC
- less getPartialCatalogObject to load a table (4->2)
- Probably needs to worry less about inconsistant metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Multiple coordinators will all load files, increasing
  Namenode pressure.

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test
M tests/webserver/test_web_pages.py
15 files changed, 143 insertions(+), 29 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/11
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 11
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#2).

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................

POC: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true, load files on
the coordinator for Iceberg tables instead of getting from
the catalogd. This is inefficient at the moment as catalogd
still loads the files but not really uses them. The long
term goal is to load only minimal info for the table on
catalogd side.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex pair. This would be
suitable for REST catalog too if the key contained
snapshot ID instead of catalog version.

Pros:
- File descs are not transferred in getPartialCatalogObject RPC
- Size of cache objects seem to decrease:
  ~543->~431 MB for 1M file table
- Plans look faster due to skipping construction of
  IcebergFileContentStore (DESCRIBE 1M file table 0.5s->1ms)
- Probably needs to worry less about inconsistant metadata
  exceptions, as the old file list remains loadable event after
  catalogd updated to a newer version.

Cons:
- Initial table loading is ~doubled as both the catalog and
  the coordinator need to load the files.
- No incremental loading, any catalog version change leads
  the reloaiding the files from scratch on coordinator side.

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
13 files changed, 161 insertions(+), 28 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/2
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13160/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Feb 2026 10:09:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13160/


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Feb 2026 14:33:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Jason Fehr, Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#12).

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................

POC IMPALA-14805: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true:
- catalogd will skip listing files while loading
  Iceberg tables
- coordinator will load the files instead and cache
  them

This assumes that local catalog mode is used, I didn't
test without it.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex + partition stats.
This would be suitable for REST catalog too if the key
contained snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak pointer to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros:
- File descs are not transferred in getPartialCatalogObject RPC
- less getPartialCatalogObject to load a table (4->2)
- Probably needs to worry less about inconsistant metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Multiple coordinators will all load files, increasing
  Namenode pressure.

Testing:
- kept load_iceberg_files_in_coordinator's default as false
  as some EE tests were not passing and I would fix them in
  a later commit
- added custom cluster test to run a subset of Iceberg tests
  with load_iceberg_files_in_coordinator=true

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test
A tests/custom_cluster/test_iceberg_load_on_coordinator.py
M tests/webserver/test_web_pages.py
16 files changed, 303 insertions(+), 26 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/12
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 12
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 7:

(7 comments)

Overall, seems like a great start.

http://gerrit.cloudera.org:8080/#/c/24049/7//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24049/7//COMMIT_MSG@15
PS7, Line 15: This assumes that local catalog mode is used, I didn't
            : test without it.
To move out of PoC, will need some validation to ensure local catalog mode is used when load_iceberg_files_in_coordinator == true


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1265
PS7, Line 1265: oon
Nit: one


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@570
PS7, Line 570:     // TODO: does it make sense to skip file loading here?
Where else would file loading be skipped if not here?


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java@169
PS7, Line 169:   public class CachedIcebergFiles {
Nit:  This class definition should be in its own file.


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java@219
PS7, Line 219:   // TODO: this doesn't really make sense
It might not right now, but in the future it could.  The `tryAllProviders` function would need to ignore `NotImplementedException` or else `DirectMetaProvider#getIcebergContentFileStore` needs to throw a `MetaProviderException`


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@7804
PS7, Line 7804:         // TODO is affectedExistingPartitions needed?
Does not seem like it is needed.


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
File fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java@142
PS7, Line 142: // TODO: different result if loadFiles=false?
It would be trivial to test by adding:

    // Test loadIcebergTable(TableMetaRef, false)
    TPartialTableInfo actualIcebergTableInfo =
        fixture.loadIcebergTable(expectedTableRef, false);
    assertThat(actualIcebergTableInfo, sameInstance(expectedIcebergTableInfo));
    verify(mockDecorated).loadIcebergTable(eq(expectedTableRef), eq(false));



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 Mar 2026 21:06:18 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 1:

(6 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24049/1/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/1/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@820
PS1, Line 820:     LOG.trace("partIds size: {} want_partition_files: {} want_partition_metadata: {} want_partition_names: {} want_partition_stats {}",
line too long (135 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1262
PS1, Line 1262:     IcebergContentFileStoreCacheKey cacheKey = new IcebergContentFileStoreCacheKey(tableRef);
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/1/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1266
PS1, Line 1266:           public Pair<IcebergContentFileStore, ListMap<TNetworkAddress>> call() throws Exception {
line too long (98 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/1/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/1/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@96
PS1, Line 96:       boolean loadFilesFromCatalog = !BackendConfig.INSTANCE.getloadIcebergFilesIncoordinator();
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/1/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@115
PS1, Line 115:             db.getCatalog().getMetaProvider().getIcebergContentFileStore(ref, icebergApiTable);
line too long (95 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/1/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
File fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java:

http://gerrit.cloudera.org:8080/#/c/24049/1/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java@142
PS1, Line 142:     when(mockDecorated.loadIcebergTable(any(TableMetaRef.class),anyBoolean())) // TODO: different result if loadFiles=false?
line too long (124 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Feb 2026 21:59:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13225/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Mar 2026 10:41:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 4: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13165/


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 00:15:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Jason Fehr, Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#17).

Change subject: IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................

IMPALA-14805: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true:
- catalogd will skip listing files while loading
  Iceberg tables
- coordinator will load the files instead and cache
  them

load_iceberg_files_in_coordinator is an experimental flag
and false by default. Most tests pass with it, but some
Iceberg DMLs still use the file list in the catalog.
IMPALA-15113 tracks fixing these.
Note that load_iceberg_files_in_coordinator=true does not
work with legacy catalog mode.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex + partition stats.
This would be suitable for REST catalog too if the key
contained snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak reference to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros of load_iceberg_files_in_coordinator=true:
- File descs are not transferred in getPartialCatalogObject RPC.
- Less getPartialCatalogObject to load a table (4->2).
- Probably needs to worry less about inconsistent metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Multiple coordinators will all load files, increasing
  Namenode pressure.

Testing:
- kept load_iceberg_files_in_coordinator's default as false
  as some EE tests were not passing
- added custom cluster test to run a subset of Iceberg tests
  with load_iceberg_files_in_coordinator=true

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A tests/custom_cluster/test_iceberg_load_on_coordinator.py
12 files changed, 291 insertions(+), 31 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/17
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 17
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14805: Allow loading Iceberg file list in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: IMPALA-14805: Allow loading Iceberg file list in coordinator
......................................................................


Patch Set 17:

ps 16/17 just rebased and updated commit message


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 17
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Jul 2026 12:56:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#7).

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................

POC IMPALA-14805: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true:
- catalogd will skip listing files while loading
  Iceberg tables
- coordinator will load the files instead and cache
  them

This assumes that local catalog mode is used, I didn't
test without it.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex + partition stats.
This would be suitable for REST catalog too if the key
contained snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak pointer to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros:
- File descs are not transferred in getPartialCatalogObject RPC
- less getPartialCatalogObject to load a table (5->3)
- Size of cache objects seem to decrease (based on JAMM):
  Before:
   543MB org.apache.impala.thrift.TPartialTableInfo
   0.5MB iceberg.BaseTable
  After:
   435MB org.apache.impala.catalog.IcebergFileContentStore
   0.5MB org.apache.iceberg.BaseTable
   <10KB org.apache.impala.thrift.TPartialTableInfo
- Plans look faster due to skipping construction of
  IcebergFileContentStore:
  DESCRIBE t: 1s->10ms
  EXPLAIN SELECT * FROM t : 3s->1.5s
  EXPLAIN SELECT * FROM t WHERE part_col=100: 1s->0.3s
- Probably needs to worry less about inconsistant metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Multiple coordinators will all load files, increasing
  Namenode pressure.

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test
M tests/webserver/test_web_pages.py
21 files changed, 366 insertions(+), 89 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/7
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 7:

(14 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@445
PS7, Line 445:     table.setIceberg_table(Utils.getTIcebergTable(this, ThriftObjectType.DESCRIPTOR_ONLY));
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@853
PS7, Line 853:     LOG.trace("partIds size: {} want_iceberg_table: {} want_partition_files: {} want_partition_metadata: {} want_partition_names: {} want_partition_stats {} want_table_constraints{}",
line too long (183 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1278
PS7, Line 1278:     IcebergContentFileStoreCacheKey cacheKey = new IcebergContentFileStoreCacheKey(tableRef);
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1284
PS7, Line 1284:               MetaProvider.CachedIcebergFiles last = lastFileStore == null ? null : lastFileStore.get();
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1285
PS7, Line 1285:               ListMap<TNetworkAddress> hostIndex = last == null ? new ListMap<>() : last.hostIndex_;
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1286
PS7, Line 1286:               IcebergContentFileStore oldFileStore = last == null ? null : last.fileStore_;
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1287
PS7, Line 1287:               return IcebergMetaProvider.getContentFileStore(tableRef, apiTable, hostIndex, oldFileStore, true);
line too long (112 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1304
PS7, Line 1304:         result.fileStore_.cloneWithoutMutables(), result.hostIndex_, result.partitionStats_);
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@600
PS7, Line 600:   public static MetaProvider.CachedIcebergFiles getContentFileStore(final TableMetaRef table,
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@611
PS7, Line 611:           oldFileStore == null ? Collections.emptyList() : oldFileStore.getPartitionList(),
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@631
PS7, Line 631:       return getContentFileStore(table, apiTable, hostIndex, null, false).fileStore_.toThrift();
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@110
PS7, Line 110:       boolean loadFilesFromCatalog = !BackendConfig.INSTANCE.getloadIcebergFilesIncoordinator();
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@121
PS7, Line 121:             db.getCatalog().getMetaProvider().getIcebergContentFileStore(ref, icebergApiTable);
line too long (95 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
File fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java@142
PS7, Line 142:     when(mockDecorated.loadIcebergTable(any(TableMetaRef.class),anyBoolean())) // TODO: different result if loadFiles=false?
line too long (124 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Mar 2026 08:29:48 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 12:

(3 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24049/12/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/12/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@473
PS12, Line 473:     table.setIceberg_table(Utils.getTIcebergTable(this, ThriftObjectType.DESCRIPTOR_ONLY));
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/12/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@933
PS12, Line 933:     LOG.trace("partIds size: {} want_iceberg_table: {} want_partition_files: {} want_partition_metadata: {} want_partition_names: {} want_partition_stats {} want_table_constraints{}",
line too long (183 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/12/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/12/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@595
PS12, Line 595:           oldFileStore == null ? Collections.emptyList() : oldFileStore.getPartitionList(),
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 12
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Jun 2026 08:26:09 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Jason Fehr, Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#13).

Change subject: WIP IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................

WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true:
- catalogd will skip listing files while loading
  Iceberg tables
- coordinator will load the files instead and cache
  them

load_iceberg_files_in_coordinator is an experimental flag
and false by default. Most tests pass with it, but some
Iceberg DMLs still use the file list in the catalog.
IMPALA-15113 tracks fixing this.
Note that load_iceberg_files_in_coordinator=true does not
work with legacy catalog mode.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex + partition stats.
This would be suitable for REST catalog too if the key
contained snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak pointer to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros:
- File descs are not transferred in getPartialCatalogObject RPC.
- Less getPartialCatalogObject to load a table (4->2).
- Probably needs to worry less about inconsistant metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Multiple coordinators will all load files, increasing
  Namenode pressure.

Testing:
- kept load_iceberg_files_in_coordinator's default as false
  as some EE tests were not passing
- added custom cluster test to run a subset of Iceberg tests
  with load_iceberg_files_in_coordinator=true

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test
A tests/custom_cluster/test_iceberg_load_on_coordinator.py
M tests/webserver/test_web_pages.py
15 files changed, 289 insertions(+), 31 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/13
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 13
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: WIP IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 13:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22958/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 13
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Jun 2026 17:30:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14805: Allow loading Iceberg file list in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Jason Fehr, Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#18).

Change subject: IMPALA-14805: Allow loading Iceberg file list in coordinator
......................................................................

IMPALA-14805: Allow loading Iceberg file list in coordinator

If load_iceberg_files_in_coordinator=true:
- catalogd will skip listing files while loading
  Iceberg tables
- coordinator will load the files instead and cache
  them

load_iceberg_files_in_coordinator is an experimental flag
and false by default. Most tests pass with it, but some
Iceberg DMLs still use the file list in the catalog.
IMPALA-15113 tracks fixing these.
Note that load_iceberg_files_in_coordinator=true does not
work with legacy catalog mode.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex + partition stats.
This would be suitable for REST catalog too if the key
contained snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak reference to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.

Pros of load_iceberg_files_in_coordinator=true:
- File descs are not transferred in getPartialCatalogObject RPC.
- Less getPartialCatalogObject to load a table (4->2).
- Probably needs to worry less about inconsistent metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Multiple coordinators will all load files, increasing
  Namenode pressure.

Testing:
- kept load_iceberg_files_in_coordinator's default as false
  as some EE tests were not passing
- added custom cluster test to run a subset of Iceberg tests
  with load_iceberg_files_in_coordinator=true

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A tests/custom_cluster/test_iceberg_load_on_coordinator.py
12 files changed, 291 insertions(+), 31 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/18
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 18
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14805: Allow loading Iceberg file list in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: IMPALA-14805: Allow loading Iceberg file list in coordinator
......................................................................


Patch Set 18:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13684/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 18
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Jul 2026 12:58:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 9: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13353/


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Apr 2026 00:09:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#9).

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................

POC IMPALA-14805: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true:
- catalogd will skip listing files while loading
  Iceberg tables
- coordinator will load the files instead and cache
  them

This assumes that local catalog mode is used, I didn't
test without it.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex + partition stats.
This would be suitable for REST catalog too if the key
contained snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak pointer to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros:
- File descs are not transferred in getPartialCatalogObject RPC
- less getPartialCatalogObject to load a table (4->2)
- Probably needs to worry less about inconsistant metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Multiple coordinators will all load files, increasing
  Namenode pressure.

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test
M tests/webserver/test_web_pages.py
15 files changed, 145 insertions(+), 30 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/9
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 3:

(6 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24049/3/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/3/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@820
PS3, Line 820:     LOG.trace("partIds size: {} want_partition_files: {} want_partition_metadata: {} want_partition_names: {} want_partition_stats {}",
line too long (135 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1262
PS3, Line 1262:     IcebergContentFileStoreCacheKey cacheKey = new IcebergContentFileStoreCacheKey(tableRef);
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/3/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1267
PS3, Line 1267:             public Pair<IcebergContentFileStore, ListMap<TNetworkAddress>> call() throws Exception {
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/3/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/3/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@96
PS3, Line 96:       boolean loadFilesFromCatalog = !BackendConfig.INSTANCE.getloadIcebergFilesIncoordinator();
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/3/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@115
PS3, Line 115:             db.getCatalog().getMetaProvider().getIcebergContentFileStore(ref, icebergApiTable);
line too long (95 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/3/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
File fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java:

http://gerrit.cloudera.org:8080/#/c/24049/3/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java@142
PS3, Line 142:     when(mockDecorated.loadIcebergTable(any(TableMetaRef.class),anyBoolean())) // TODO: different result if loadFiles=false?
line too long (124 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Feb 2026 10:10:57 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 11:

(3 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24049/11/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/11/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@473
PS11, Line 473:     table.setIceberg_table(Utils.getTIcebergTable(this, ThriftObjectType.DESCRIPTOR_ONLY));
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/11/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@933
PS11, Line 933:     LOG.trace("partIds size: {} want_iceberg_table: {} want_partition_files: {} want_partition_metadata: {} want_partition_names: {} want_partition_stats {} want_table_constraints{}",
line too long (183 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/11/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/11/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@595
PS11, Line 595:           oldFileStore == null ? Collections.emptyList() : oldFileStore.getPartitionList(),
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 11
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 20 Jun 2026 10:53:14 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 10:

PS 10 was a rebase to include https://gerrit.cloudera.org/#/c/24210/
The actual incremental loading optimization is not integrated yet for the loading in coordinator case


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 May 2026 08:13:02 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14805: Allow loading Iceberg file list in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: IMPALA-14805: Allow loading Iceberg file list in coordinator
......................................................................


Patch Set 17:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/23060/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 17
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Jul 2026 13:19:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 12:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22953/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 12
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Jun 2026 08:46:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13217/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Mar 2026 17:01:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 7: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13229/


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Mar 2026 13:00:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 8:

(4 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24049/8/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/8/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@446
PS8, Line 446:     table.setIceberg_table(Utils.getTIcebergTable(this, ThriftObjectType.DESCRIPTOR_ONLY));
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/8/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@858
PS8, Line 858:     LOG.trace("partIds size: {} want_iceberg_table: {} want_partition_files: {} want_partition_metadata: {} want_partition_names: {} want_partition_stats {} want_table_constraints{}",
line too long (183 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/8/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/8/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@597
PS8, Line 597:           oldFileStore == null ? Collections.emptyList() : oldFileStore.getPartitionList(),
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/8/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
File fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java:

http://gerrit.cloudera.org:8080/#/c/24049/8/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java@141
PS8, Line 141:     when(mockDecorated.loadIcebergTable(any(TableMetaRef.class))) // TODO: different result if loadFiles=false?
line too long (111 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 20 Apr 2026 19:42:00 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 9:

(3 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@446
PS9, Line 446:     table.setIceberg_table(Utils.getTIcebergTable(this, ThriftObjectType.DESCRIPTOR_ONLY));
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@858
PS9, Line 858:     LOG.trace("partIds size: {} want_iceberg_table: {} want_partition_files: {} want_partition_metadata: {} want_partition_names: {} want_partition_stats {} want_table_constraints{}",
line too long (183 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@597
PS9, Line 597:           oldFileStore == null ? Collections.emptyList() : oldFileStore.getPartitionList(),
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 20 Apr 2026 19:44:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 12: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 12
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Jun 2026 12:35:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: WIP IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 14:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22959/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 14
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Jun 2026 17:44:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: WIP IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 16: Code-Review+1

LGTM!


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 16
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Jul 2026 09:59:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#4).

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................

POC: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true, load files on
the coordinator for Iceberg tables instead of getting from
the catalogd. This is inefficient at the moment as catalogd
still loads the files but not really uses them. The long
term goal is to load only minimal info for the table on
catalogd side.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex pair. This would be
suitable for REST catalog too if the key contained
snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak pointer to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros:
- File descs are not transferred in getPartialCatalogObject RPC
- Size of cache objects seem to decrease:
  Before:
   543 org.apache.impala.thrift.TPartialTableInfo
   0.5MB iceberg.BaseTable
  After
   431 org.apache.impala.catalog.IcebergFileContentStore
   3MB org.apache.impala.thrift.TPartialTableInfo
   0.5MB org.apache.iceberg.BaseTable
- Plans look faster due to skipping construction of
  IcebergFileContentStore:
  DESCRIBE t: 1s->10ms
  EXPLAIN SELECT * FROM t : 3s->1.5s
  EXPLAIN SELECT * FROM t WHERE part_col=100: 1s->0.3s
- Probably needs to worry less about inconsistant metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Initial table loading is ~doubled as both the catalog and
  the coordinator need to load the files.
- Multiple coordinators will all load files, increasing
  Namenode pressure.

With the exception of the last one all cons seem solvable.

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
14 files changed, 245 insertions(+), 39 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/4
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 2: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13156/


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Feb 2026 02:24:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: WIP IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 16:

(2 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24049/16/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/16/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1396
PS16, Line 1396:               // requests lead to writing an older file store than the one for the freshest
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/16/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1397
PS16, Line 1397:               // catalog version, most files are probably still reusable and the next "fresh"
line too long (93 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 16
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Jun 2026 12:17:24 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#5).

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................

POC IMPALA-14805: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true:
- catalogd will skip listing files while loading
  Iceberg tables
- coordinator will load the files instead and cache
  them

This assumes that local catalog mode is used, I didn't
test without it.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex + partition stats.
This would be suitable for REST catalog too if the key
contained snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak pointer to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros:
- File descs are not transferred in getPartialCatalogObject RPC
- less getPartialCatalogObject to load a table (5->3)
- Size of cache objects seem to decrease (based on JAMM):
  Before:
   543MB org.apache.impala.thrift.TPartialTableInfo
   0.5MB iceberg.BaseTable
  After:
   435MB org.apache.impala.catalog.IcebergFileContentStore
   0.5MB org.apache.iceberg.BaseTable
   <10KB org.apache.impala.thrift.TPartialTableInfo
- Plans look faster due to skipping construction of
  IcebergFileContentStore:
  DESCRIBE t: 1s->10ms
  EXPLAIN SELECT * FROM t : 3s->1.5s
  EXPLAIN SELECT * FROM t WHERE part_col=100: 1s->0.3s
- Probably needs to worry less about inconsistant metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Multiple coordinators will all load files, increasing
  Namenode pressure.

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test
20 files changed, 359 insertions(+), 86 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/5
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 12:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13646/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 12
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Jun 2026 08:25:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 11: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13645/


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 11
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 20 Jun 2026 17:02:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14805: Allow loading Iceberg file list in coordinator

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: IMPALA-14805: Allow loading Iceberg file list in coordinator
......................................................................


Patch Set 18: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/24049/15/tests/custom_cluster/test_iceberg_load_on_coordinator.py
File tests/custom_cluster/test_iceberg_load_on_coordinator.py:

http://gerrit.cloudera.org:8080/#/c/24049/15/tests/custom_cluster/test_iceberg_load_on_coordinator.py@32
PS15, Line 32:   # TODO: think through what tests to run in with special config.
> Measured the tests. These can be run in parallel as all tests need the same
Yes, our test infra doesn't run custom cluster tests in parallel. With current sharding, the 5 minutes redcues to 2-3 minutes, not sure how bad is that.



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 18
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Jul 2026 11:36:55 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 10:

(3 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24049/10/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/10/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@461
PS10, Line 461:     table.setIceberg_table(Utils.getTIcebergTable(this, ThriftObjectType.DESCRIPTOR_ONLY));
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/10/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@943
PS10, Line 943:     LOG.trace("partIds size: {} want_iceberg_table: {} want_partition_files: {} want_partition_metadata: {} want_partition_names: {} want_partition_stats {} want_table_constraints{}",
line too long (183 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/10/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/10/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@598
PS10, Line 598:           oldFileStore == null ? Collections.emptyList() : oldFileStore.getPartitionList(),
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 May 2026 08:12:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 10:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/22321/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 May 2026 08:33:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Jason Fehr, Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#8).

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................

POC IMPALA-14805: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true:
- catalogd will skip listing files while loading
  Iceberg tables
- coordinator will load the files instead and cache
  them

This assumes that local catalog mode is used, I didn't
test without it.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex + partition stats.
This would be suitable for REST catalog too if the key
contained snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak pointer to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros:
- File descs are not transferred in getPartialCatalogObject RPC
- less getPartialCatalogObject to load a table (4->2)
- Probably needs to worry less about inconsistant metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Multiple coordinators will all load files, increasing
  Namenode pressure.

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test
M tests/webserver/test_web_pages.py
16 files changed, 146 insertions(+), 31 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/8
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22183/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 20 Apr 2026 20:14:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Jason Fehr, Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#14).

Change subject: WIP IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................

WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true:
- catalogd will skip listing files while loading
  Iceberg tables
- coordinator will load the files instead and cache
  them

load_iceberg_files_in_coordinator is an experimental flag
and false by default. Most tests pass with it, but some
Iceberg DMLs still use the file list in the catalog.
IMPALA-15113 tracks fixing these.
Note that load_iceberg_files_in_coordinator=true does not
work with legacy catalog mode.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex + partition stats.
This would be suitable for REST catalog too if the key
contained snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak reference to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros of load_iceberg_files_in_coordinator=true:
- File descs are not transferred in getPartialCatalogObject RPC.
- Less getPartialCatalogObject to load a table (4->2).
- Probably needs to worry less about inconsistant metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Multiple coordinators will all load files, increasing
  Namenode pressure.

Testing:
- kept load_iceberg_files_in_coordinator's default as false
  as some EE tests were not passing
- added custom cluster test to run a subset of Iceberg tests
  with load_iceberg_files_in_coordinator=true

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A tests/custom_cluster/test_iceberg_load_on_coordinator.py
12 files changed, 285 insertions(+), 31 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/14
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 14
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 11:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13645/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 11
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 20 Jun 2026 10:52:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21811/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Feb 2026 22:24:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21934/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Mar 2026 08:52:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 5: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13217/


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Mar 2026 22:25:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: WIP IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 16:

(6 comments)

PS 16 fixed Zoltan's comments and also did a rebase, which included the ehcache removal (IMPALA-15114)

http://gerrit.cloudera.org:8080/#/c/24049/15//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24049/15//COMMIT_MSG@19
PS15, Line 19: load_iceberg_files_in_coordinator=true does not
             : work with legacy catalog mode
> We could check incompatible flag combinations at startup.
The flags are experimental, I would put effort in this once there will be flags usable in production. Preferably it would be enough to set this in coordinator or catalogd, just didn't decide which one.


http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1364
PS9, Line 1364: etaProvider.CachedIcebergFiles>() {
> As Impala bumped to Java 17, I plan to remove Ehcache. Created IMPALA-15114
IMPALA-15114 was merged and ehcache was removed.


http://gerrit.cloudera.org:8080/#/c/24049/15/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/15/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1354
PS15, Line 1354:     org.apache.iceberg.Table apiTable =
               :         loadIcebergApiTable(tableRef, tableParams, tableRef.msTable_);
               : 
               :     String itemStr = "content file store for " + tableRef.dbName_ + "."
               :         + tableRef.tableName_;
               :     // TODO: use snapshot id for key to make usable for REST catalog
               :     IcebergContentFileStoreCacheKey cacheKey =
               :         new IcebergContentFileStoreCacheKey(tableRef);
               :     MetaProvider.CachedIcebergFiles result =
               :         loadWithCaching(itemStr, TABLE_METADATA_CACHE_CATEGORY, cacheKey,
               :           new Callable<MetaProvider.CachedIcebergFiles>() {
               :             @Override
               :             public MetaProvider.CachedIcebergFiles call() throws Exception {
               :               // Try to lookup the last stored IcebergContentFileStore to help with
               :               // incremental loading. A weak reference is used to allow it to be GCd
               :               // if it gets evicted from cache. The key doesn't include catalog version,
               :               // so there is always only o
> This is not needed when the files are in the cache. Would it be worth addin
Good idea, done.
I was worried about this because it is not legal to call cache functions in the cache's functions like compute(), but the logic below does not run within compute()


http://gerrit.cloudera.org:8080/#/c/24049/15/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/15/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@599
PS15, Line 599:  
> nit: missing space
Done


http://gerrit.cloudera.org:8080/#/c/24049/15/fe/src/main/java/org/apache/impala/service/BackendConfig.java
File fe/src/main/java/org/apache/impala/service/BackendConfig.java:

http://gerrit.cloudera.org:8080/#/c/24049/15/fe/src/main/java/org/apache/impala/service/BackendConfig.java@660
PS15, Line 660: InCoordinator
> nit: InCoordinator
Done


http://gerrit.cloudera.org:8080/#/c/24049/15/tests/custom_cluster/test_iceberg_load_on_coordinator.py
File tests/custom_cluster/test_iceberg_load_on_coordinator.py:

http://gerrit.cloudera.org:8080/#/c/24049/15/tests/custom_cluster/test_iceberg_load_on_coordinator.py@32
PS15, Line 32:   # TODO: think through what tests to run in with special config.
> How much time does it take to run these tests?
Measured the tests. These can be run in parallel as all tests need the same custom cluster. Maybe Impala's test suites don't do that for custom cluster tests?

time impala-py.test tests/custom_cluster/test_iceberg_load_on_coordinator.py 

real	4m59.816s

time impala-py.test -n6 tests/custom_cluster/test_iceberg_load_on_coordinator.py
real	1m40.950s



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 16
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Jun 2026 12:23:46 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Jason Fehr, Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#16).

Change subject: WIP IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................

WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true:
- catalogd will skip listing files while loading
  Iceberg tables
- coordinator will load the files instead and cache
  them

load_iceberg_files_in_coordinator is an experimental flag
and false by default. Most tests pass with it, but some
Iceberg DMLs still use the file list in the catalog.
IMPALA-15113 tracks fixing these.
Note that load_iceberg_files_in_coordinator=true does not
work with legacy catalog mode.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex + partition stats.
This would be suitable for REST catalog too if the key
contained snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak reference to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros of load_iceberg_files_in_coordinator=true:
- File descs are not transferred in getPartialCatalogObject RPC.
- Less getPartialCatalogObject to load a table (4->2).
- Probably needs to worry less about inconsistent metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Multiple coordinators will all load files, increasing
  Namenode pressure.

Testing:
- kept load_iceberg_files_in_coordinator's default as false
  as some EE tests were not passing
- added custom cluster test to run a subset of Iceberg tests
  with load_iceberg_files_in_coordinator=true

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A tests/custom_cluster/test_iceberg_load_on_coordinator.py
12 files changed, 291 insertions(+), 31 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/16
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 16
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14805: Allow loading Iceberg file list in coordinator

Posted by "Noemi Pap-Takacs (Code Review)" <ge...@cloudera.org>.
Noemi Pap-Takacs has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: IMPALA-14805: Allow loading Iceberg file list in coordinator
......................................................................


Patch Set 18: Code-Review+1

(1 comment)

Thanks for implementing this!
I think coordinator-only loading is the good direction for Iceberg tables - but we might need to do some refactoring in the future.

http://gerrit.cloudera.org:8080/#/c/24049/18/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/18/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@598
PS18, Line 598: IcebergFileMetadataLoader iceFml = new IcebergFileMetadataLoader(
              :           apiTable, oldFiles, hostIndex, groupedFiles, oldPartitions, false, null);
This will always be a full load without optimization, e.g. incremental load. This could be a regression compared to the catalog loading.
Do you plan to add incremental load later?



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 18
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 15 Jul 2026 14:43:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13229/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Mar 2026 08:42:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13165/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Feb 2026 19:58:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 4:

(11 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@820
PS4, Line 820:     LOG.trace("partIds size: {} want_partition_files: {} want_partition_metadata: {} want_partition_names: {} want_partition_stats {}",
line too long (135 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1277
PS4, Line 1277:     IcebergContentFileStoreCacheKey cacheKey = new IcebergContentFileStoreCacheKey(tableRef);
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1282
PS4, Line 1282:             public Pair<IcebergContentFileStore, ListMap<TNetworkAddress>> call() throws Exception {
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1283
PS4, Line 1283:               Pair<IcebergContentFileStore, ListMap<TNetworkAddress>> last = lastFileStore == null ? null : lastFileStore.get();
line too long (128 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1284
PS4, Line 1284:               ListMap<TNetworkAddress> hostIndex = last == null ? new ListMap<>() : last.getSecond();
line too long (101 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1285
PS4, Line 1285:               IcebergContentFileStore oldFileStore = last == null ? null : last.getFirst();
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1287
PS4, Line 1287:                   IcebergMetaProvider.getContentFileStore(tableRef, apiTable, hostIndex, oldFileStore);
line too long (103 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@598
PS4, Line 598:           oldFileStore == null ? Collections.emptyList() : oldFileStore.getPartitionList(),
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@96
PS4, Line 96:       boolean loadFilesFromCatalog = !BackendConfig.INSTANCE.getloadIcebergFilesIncoordinator();
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@115
PS4, Line 115:             db.getCatalog().getMetaProvider().getIcebergContentFileStore(ref, icebergApiTable);
line too long (95 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
File fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java:

http://gerrit.cloudera.org:8080/#/c/24049/4/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java@142
PS4, Line 142:     when(mockDecorated.loadIcebergTable(any(TableMetaRef.class),anyBoolean())) // TODO: different result if loadFiles=false?
line too long (124 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Feb 2026 20:00:24 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21820/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Feb 2026 20:25:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14805: Allow loading Iceberg file list in coordinator

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: IMPALA-14805: Allow loading Iceberg file list in coordinator
......................................................................


Patch Set 18:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/24049/18/be/src/util/backend-gflag-util.cc
File be/src/util/backend-gflag-util.cc:

http://gerrit.cloudera.org:8080/#/c/24049/18/be/src/util/backend-gflag-util.cc@332
PS18, Line 332:     "are loaded locally instead of requesting from the catalogd.");
nit: Let's mention whether this should be set in catalogd.


http://gerrit.cloudera.org:8080/#/c/24049/17/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/17/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1397
PS17, Line 1397:               // catalog version, most files are probably still reusable and the next "fresh"
> line too long (93 > 90)
Please resolve these long lines


http://gerrit.cloudera.org:8080/#/c/24049/18/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/18/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1363
PS18, Line 1363: TABLE_METADATA_CACHE_CATEGORY
Can we use a new category? This is currently used for fetching hmsTable and apiTable from catalogd. It's also used in fetching IcebergContentFileStore from catalogd when the flag is false. But I think that should also use a new category, e.g. ICEBERG_FILE_METADATA_CACHE_CATEGORY = "IcebergFiles".

With a new category we can add tests verifying this code path is exercised. It also improves diagnostics.


http://gerrit.cloudera.org:8080/#/c/24049/18/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1399
PS18, Line 1399: WeakReference
nit: WeakReference<>


http://gerrit.cloudera.org:8080/#/c/24049/18/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/18/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@598
PS18, Line 598: IcebergFileMetadataLoader iceFml = new IcebergFileMetadataLoader(
              :           apiTable, oldFiles, hostIndex, groupedFiles, oldPartitions, false, null);
> This will always be a full load without optimization, e.g. incremental load
Does it mean we can load files with predicates from the query that triggers the load?


http://gerrit.cloudera.org:8080/#/c/24049/18/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/24049/18/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@7946
PS18, Line 7946:       FeFsPartition partition,
nit: merge this with the previous line



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 18
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 17 Jul 2026 05:41:21 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: WIP IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 15:

(5 comments)

Nice patch! Only found minor issues.

http://gerrit.cloudera.org:8080/#/c/24049/15//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24049/15//COMMIT_MSG@19
PS15, Line 19: load_iceberg_files_in_coordinator=true does not
             : work with legacy catalog mode
We could check incompatible flag combinations at startup.


http://gerrit.cloudera.org:8080/#/c/24049/15/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/15/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1354
PS15, Line 1354:     // TODO: rethink how to get api table
               :     TableParams tableParams = new TableParams(tableRef.msTable_);
               :     org.apache.iceberg.Table apiTable =
               :         loadIcebergApiTable(tableRef, tableParams, tableRef.msTable_);
               : 
               :     // Try to lookup the last stored IcebergContentFileStore to help with
               :     // incremental loading. A weak reference is used to allow it to be GCd
               :     // if it gets evicted from cache. The key doesn't include catalog version,
               :     // so there is always only one lastFileStore per table.
               :     // The weak reference and key have non-zero weight, so the entry can be evicted
               :     // by weight limit. The weigher does not go through weak references, so the
               :     // backing file store won't be double counted.
               :     LastIcebergContentFileStoreCacheKey tableKey =
               :         new LastIcebergContentFileStoreCacheKey(tableRef);
               :     WeakReference<MetaProvider.CachedIcebergFiles> lastFileStore =
               :         (WeakReference<MetaProvider.CachedIcebergFiles>)
               :             cache_.getIfPresent(tableKey);
This is not needed when the files are in the cache. Would it be worth adding a quick getIfPresent(cacheKey) check? Or maybe they could be moved inside call() via some extra indirection.


http://gerrit.cloudera.org:8080/#/c/24049/15/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/15/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@599
PS15, Line 599: o
nit: missing space


http://gerrit.cloudera.org:8080/#/c/24049/15/fe/src/main/java/org/apache/impala/service/BackendConfig.java
File fe/src/main/java/org/apache/impala/service/BackendConfig.java:

http://gerrit.cloudera.org:8080/#/c/24049/15/fe/src/main/java/org/apache/impala/service/BackendConfig.java@660
PS15, Line 660: Incoordinator
nit: InCoordinator


http://gerrit.cloudera.org:8080/#/c/24049/15/tests/custom_cluster/test_iceberg_load_on_coordinator.py
File tests/custom_cluster/test_iceberg_load_on_coordinator.py:

http://gerrit.cloudera.org:8080/#/c/24049/15/tests/custom_cluster/test_iceberg_load_on_coordinator.py@32
PS15, Line 32:   # TODO: think through what tests to run in with special config.
How much time does it take to run these tests?
Since the tests are running sequentially, maybe we should only run them in exhaustive mode, or merge a few together?



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Jun 2026 15:39:00 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 17:

(2 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24049/17/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/17/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1396
PS17, Line 1396:               // requests lead to writing an older file store than the one for the freshest
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/17/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1397
PS17, Line 1397:               // catalog version, most files are probably still reusable and the next "fresh"
line too long (93 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 17
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Jul 2026 12:55:54 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14805: Allow loading Iceberg file list in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: IMPALA-14805: Allow loading Iceberg file list in coordinator
......................................................................


Patch Set 18: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 18
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 01 Jul 2026 17:26:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Jason Fehr, Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#10).

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................

POC IMPALA-14805: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true:
- catalogd will skip listing files while loading
  Iceberg tables
- coordinator will load the files instead and cache
  them

This assumes that local catalog mode is used, I didn't
test without it.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex + partition stats.
This would be suitable for REST catalog too if the key
contained snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak pointer to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.
TODO: optimization from IMPALA-14804 not integrated yet

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros:
- File descs are not transferred in getPartialCatalogObject RPC
- less getPartialCatalogObject to load a table (4->2)
- Probably needs to worry less about inconsistant metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Multiple coordinators will all load files, increasing
  Namenode pressure.

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-mixed-format-position-deletes.test
M tests/webserver/test_web_pages.py
15 files changed, 146 insertions(+), 30 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/10
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 10
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22182/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 20 Apr 2026 20:14:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/21814/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Feb 2026 10:34:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13156/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Feb 2026 22:01:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#3).

Change subject: POC: LocalIcebergTable loads files in coordinator
......................................................................

POC: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true, load files on
the coordinator for Iceberg tables instead of getting from
the catalogd. This is inefficient at the moment as catalogd
still loads the files but not really uses them. The long
term goal is to load only minimal info for the table on
catalogd side.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex pair. This would be
suitable for REST catalog too if the key contained
snapshot ID instead of catalog version.

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros:
- File descs are not transferred in getPartialCatalogObject RPC
- Size of cache objects seem to decrease:
  Before:
   543 org.apache.impala.thrift.TPartialTableInfo
   0.5MB iceberg.BaseTable
  After
   431 org.apache.impala.catalog.IcebergFileContentStore
   3MB org.apache.impala.thrift.TPartialTableInfo
   0.5MB org.apache.iceberg.BaseTable
- Plans look faster due to skipping construction of
  IcebergFileContentStore:
  DESCRIBE t: 1s->10ms
  EXPLAIN SELECT * FROM t : 3s->1.5s
  EXPLAIN SELECT * FROM t WHERE part_col=100: 1s->0.3s
- Probably needs to worry less about inconsistant metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Initial table loading is ~doubled as both the catalog and
  the coordinator need to load the files.
- No incremental loading, any catalog version change leads
  the reloaiding the files from scratch on coordinator side.
- Multiple coordinators will all load files, increasing
  Namenode pressure.

With the exception of the last one all cons seem solvable.

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
14 files changed, 205 insertions(+), 36 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/3
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 7:

(12 comments)

Nice change! And sorry for the late response!

http://gerrit.cloudera.org:8080/#/c/24049/7//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24049/7//COMMIT_MSG@24
PS7, Line 24: weak pointer
nit: weak reference


http://gerrit.cloudera.org:8080/#/c/24049/7//COMMIT_MSG@49
PS7, Line 49: inconsistant
nit: inconsistent


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
File fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@162
PS7, Line 162:   // The following members are considered immutable and shouldn't change after
             :   // constructing the object:
At some places we have BEGIN / END sections to make it more explicit. See e.g. StatementBase.java, parquet-column-readers.h


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@231
PS7, Line 231: new IcebergContentFileStore();
I wonder if Java is smart enough to skip the creation of the empty containers. Alternatively, we could introduce a new constructor that takes every immutable member.


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@886
PS7, Line 886: prefixes
nit: Could be just 'List.of(getLocation())'


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1257
PS7, Line 1257: getIcebergContentFileStore(final TableMetaRef table,
nit: fits earlier line


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1263
PS7, Line 1263: do
nit: to


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1271
PS7, Line 1271:     WeakReference<MetaProvider.CachedIcebergFiles> lastFileStore =
              :         (WeakReference<MetaProvider.CachedIcebergFiles>)
              :             cache_.getIfPresent(tableKey);
Is it possible to move this inside the call() method?


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1299
PS7, Line 1299:     // Mutable concurrent hash maps are used to cache files from time-travel queries,
              :     // which is thread-safe, but would increase the size of the cached object without
              :     // updating weight.
              :     // TODO: come up with a way to cache files in time-travel queries
I wonder if it even make sense to store time-travel FDs and partitions in the IcebergContentFileStore, since they are not being re-used anyway (except maybe in legacy catalog mode).


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@138
PS7, Line 138: isInIcebergTable_
Is it necessary to have this? The internal FS table should be minimal since IMPALA-13737.


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@98
PS7, Line 98: loadedInCatalog_
It is only written, never read.


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@158
PS7, Line 158:     // TODO: understand if this is needed with local loading
LocalTable.load() invokes t.loadColumnStats() -> metaProvider.loadTableColumnStatistics() -> getPartialCatalogObject() after loading the table. Since file loading happens between the first getPartialCatalogObject() and the last, I'd assume we still need to warmup the cache. Or it would be nice to avoid the CatalogD interaction in this case.



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Mar 2026 14:08:52 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 6: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13225/


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Mar 2026 16:18:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 5:

(13 comments)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@828
PS5, Line 828:     LOG.trace("partIds size: {} want_iceberg_table: {} want_partition_files: {} want_partition_metadata: {} want_partition_names: {} want_partition_stats {} want_table_constraints{}",
line too long (183 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1278
PS5, Line 1278:     IcebergContentFileStoreCacheKey cacheKey = new IcebergContentFileStoreCacheKey(tableRef);
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1284
PS5, Line 1284:               MetaProvider.CachedIcebergFiles last = lastFileStore == null ? null : lastFileStore.get();
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1285
PS5, Line 1285:               ListMap<TNetworkAddress> hostIndex = last == null ? new ListMap<>() : last.hostIndex_;
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1286
PS5, Line 1286:               IcebergContentFileStore oldFileStore = last == null ? null : last.fileStore_;
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1287
PS5, Line 1287:               return IcebergMetaProvider.getContentFileStore(tableRef, apiTable, hostIndex, oldFileStore, true);
line too long (112 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1304
PS5, Line 1304:         result.fileStore_.cloneWithoutMutables(), result.hostIndex_, result.partitionStats_);
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@599
PS5, Line 599:   public static MetaProvider.CachedIcebergFiles getContentFileStore(final TableMetaRef table,
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@610
PS5, Line 610:           oldFileStore == null ? Collections.emptyList() : oldFileStore.getPartitionList(),
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java@630
PS5, Line 630:       return getContentFileStore(table, apiTable, hostIndex, null, false).fileStore_.toThrift();
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@106
PS5, Line 106:       boolean loadFilesFromCatalog = !BackendConfig.INSTANCE.getloadIcebergFilesIncoordinator();
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@117
PS5, Line 117:             db.getCatalog().getMetaProvider().getIcebergContentFileStore(ref, icebergApiTable);
line too long (95 > 90)


http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java
File fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java:

http://gerrit.cloudera.org:8080/#/c/24049/5/fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java@142
PS5, Line 142:     when(mockDecorated.loadIcebergTable(any(TableMetaRef.class),anyBoolean())) // TODO: different result if loadFiles=false?
line too long (124 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Mar 2026 17:03:03 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 9:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1250
PS9, Line 1250: if(
nit: need a space


http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1360
PS9, Line 1360: do be GCd
nit: "to be GCed"


http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1362
PS9, Line 1362: oon
nit: "one"


http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1364
PS9, Line 1364: The weigher does not go through weak references
Is it true for Ehcache SizeOf? I don't see code handling this in https://github.com/ehcache/sizeof
We might need a custom SizeOfFilter to ignore WeakReference.
https://github.com/ehcache/sizeof/blob/master/src/main/java/org/ehcache/sizeof/filters/SizeOfFilter.java

We currently use jamm for Java 15+. For older java versions we are still using sizeof.


http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/service/BackendConfig.java
File fe/src/main/java/org/apache/impala/service/BackendConfig.java:

http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/service/BackendConfig.java@640
PS9, Line 640: getloadIcebergFilesIncoordinator
nit: "getLoadIcebergFilesInCoordinator"



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 9
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 21 Apr 2026 11:30:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] POC IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: POC IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 11:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/22947/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 11
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Sat, 20 Jun 2026 11:13:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Jason Fehr, Zoltan Borok-Nagy, Noemi Pap-Takacs, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/24049

to look at the new patch set (#15).

Change subject: WIP IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................

WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

If load_iceberg_files_in_coordinator=true:
- catalogd will skip listing files while loading
  Iceberg tables
- coordinator will load the files instead and cache
  them

load_iceberg_files_in_coordinator is an experimental flag
and false by default. Most tests pass with it, but some
Iceberg DMLs still use the file list in the catalog.
IMPALA-15113 tracks fixing these.
Note that load_iceberg_files_in_coordinator=true does not
work with legacy catalog mode.

Instead of caching TPartialTableInfo this solution caches
IcebergFileContentStore + hostIndex + partition stats.
This would be suitable for REST catalog too if the key
contained snapshot ID instead of catalog version.

Incremental table loading is implemented by storing a
weak reference to the last used IcebergFileContentStore for
each table in the cache. When a new IcebergFileContentStore
is requested for the table, this weak pointer is looked
up and if found, the old partition and files lists are
reused, similarly to existing implementation in catalogd.

Examples are for 1M files, 25K partitions Iceberg table
on my dev machine.

Pros of load_iceberg_files_in_coordinator=true:
- File descs are not transferred in getPartialCatalogObject RPC.
- Less getPartialCatalogObject to load a table (4->2).
- Probably needs to worry less about inconsistent metadata
  exceptions, as the old file list remains loadable even after
  catalogd updated to a newer version.

Cons:
- Multiple coordinators will all load files, increasing
  Namenode pressure.

Testing:
- kept load_iceberg_files_in_coordinator's default as false
  as some EE tests were not passing
- added custom cluster test to run a subset of Iceberg tests
  with load_iceberg_files_in_coordinator=true

Change-Id: I6732af76a2e040fa57e39260302951466037b934
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A tests/custom_cluster/test_iceberg_load_on_coordinator.py
12 files changed, 285 insertions(+), 31 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/24049/15
-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: WIP IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 15:

(14 comments)

http://gerrit.cloudera.org:8080/#/c/24049/7//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/24049/7//COMMIT_MSG@24
PS7, Line 24: This would b
> nit: weak reference
Done


http://gerrit.cloudera.org:8080/#/c/24049/7//COMMIT_MSG@49
PS7, Line 49: tor's defaul
> nit: inconsistent
Done


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
File fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@162
PS7, Line 162:      * @param maxFiles Maximum files to include
             :      * @return Thrift map for
> At some places we have BEGIN / END sections to make it more explicit. See e
Done


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1257
PS7, Line 1257: ebergContentFileStoreCacheKey cacheKey =
> nit: fits earlier line
Done


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1263
PS7, Line 1263: 
> nit: to
Done


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1265
PS7, Line 1265: ddr
> Nit: one
Done


http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1250
PS9, Line 1250: if 
> nit: need a space
Done


http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1360
PS9, Line 1360: to be GCd
> nit: "to be GCed"
Done


http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1362
PS9, Line 1362: one
> nit: "one"
Done


http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1364
PS9, Line 1364: The weigher does not go through weak references
> Is it true for Ehcache SizeOf? I don't see code handling this in https://gi
As Impala bumped to Java 17, I plan to remove Ehcache. Created IMPALA-15114


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@138
PS7, Line 138: chema if it's ext
> Is it necessary to have this? The internal FS table should be minimal since
As removed in later PS.


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java:

http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@98
PS7, Line 98: 
> It is only written, never read.
Was removed in later patch.


http://gerrit.cloudera.org:8080/#/c/24049/7/fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java@158
PS7, Line 158:     tableParams_ = tableParams;
> LocalTable.load() invokes t.loadColumnStats() -> metaProvider.loadTableColu
I kept the warmup for now - the risk of updating the table while the file list is loaded and making the old columns stats unreachable is still there.


http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/service/BackendConfig.java
File fe/src/main/java/org/apache/impala/service/BackendConfig.java:

http://gerrit.cloudera.org:8080/#/c/24049/9/fe/src/main/java/org/apache/impala/service/BackendConfig.java@640
PS9, Line 640: isOtelTraceEnabled() {
> nit: "getLoadIcebergFilesInCoordinator"
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 15
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 Jun 2026 17:36:32 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] WIP IMPALA-14805: LocalIcebergTable loads files in coordinator

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24049 )

Change subject: WIP IMPALA-14805: LocalIcebergTable loads files in coordinator
......................................................................


Patch Set 16:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/23055/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/24049
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6732af76a2e040fa57e39260302951466037b934
Gerrit-Change-Number: 24049
Gerrit-PatchSet: 16
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Jun 2026 12:37:00 +0000
Gerrit-HasComments: No