You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2026/02/27 17:25:00 UTC
[Impala-ASF-CR] POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24052
Change subject: POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
......................................................................
POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
Quick and dirty solution to speed up IcebergFileMetadataLoader,
I am not sure if it is correct.
When working with a ~1M file Iceberg table, noticed that
incremental loading that loads 0 files (e.g. altering a table
property) takes nearly as much time as a full table load. Based
on some jstacks hadoop.fs.Path functions were identified as the
main CPU consumers. Tried replacing them with quick variants
that assume that the parsed URL as correct and fully qualified.
Note sure what can we expect from paths in Iceberg, but I assume
that their validity should be checked only once at most.
Full load: 13s->10s
Incremental load (0 files): 9s->3.5s
Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
---
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
2 files changed, 75 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/24052/1
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 7: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13203/
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Mar 2026 13:25:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13203/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Mar 2026 09:07:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21847/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2026 14:35:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Quick and dirty solution to speed up IcebergFileMetadataLoader.
Its correctness is based on the assumption that Iceberg file
locations must be normalized.
Noticed in flamegraphs that org.apache.hadoop.fs.Path constructor
is one of the main CPU consumers during Iceberg table loading,
especially incremental reloads when most file descriptors are reused.
hadoop.fs.Path was used to relativize locations compared to base
table location and to get the "path" part of the URI. These can
be done with simple String operations if we can assume that the
URIs are normalized.
Results on 1M file 25K partition Iceberg table:
Full load: 13s->10s
Incremental load (0 files): 9s->3.5s
hadoop.fs.Path constructor still uses significant CPU time after
the change, but mainly in functions that run in parallel, so
its effect is not longer that visible in total execution time.
See Jira for before/after flamegraphs.
Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Reviewed-on: http://gerrit.cloudera.org:8080/24052
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Csaba Ringhofer <cs...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
3 files changed, 65 insertions(+), 25 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved
Csaba Ringhofer: Verified
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21866/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 14:02:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
......................................................................
Patch Set 1: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13163/
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Feb 2026 21:52:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13170/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 16:21:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Mar 2026 08:57:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 20:38:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21848/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2026 14:46:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 4:
(3 comments)
Thanks for looking into it, nice speedup!
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
File fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java:
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@193
PS4, Line 193: iceApiTable.location()
BaseTable.location() is not a simple getter, it should be moved outside the loop.
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@365
PS4, Line 365: use slow path.
We could log the URI here. Maybe with trace-level only, in case there are lots of unexpected URIs.
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@355
PS4, Line 355: path.length() >= tablePathStr_.length() + 1
nit: path.length() > tablePathStr_.length()
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2026 20:41:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
......................................................................
Patch Set 1:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24052/1/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/24052/1/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@367
PS1, Line 367: if (contentFilePathStr.startsWith(tablePathStr_) && contentFilePathStr.charAt(tablePathStr_.length()) == '/') {
line too long (115 > 90)
http://gerrit.cloudera.org:8080/#/c/24052/1/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@379
PS1, Line 379: throw new TableLoadingException(String.format("Failed to load Iceberg datafile " +
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Feb 2026 17:26:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has removed a vote on this change.
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Removed Verified-1 by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has removed a vote on this change.
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Removed -Verified by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24052
to look at the new patch set (#4).
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Quick and dirty solution to speed up IcebergFileMetadataLoader.
Its correctness is based on the assumption that Iceberg file
locations must be normalized.
Noticed in flamegraphs that org.apache.hadoop.fs.Path constructor
is one of the main CPU consumers during Iceberg table loading,
especially incremental reloads when most file descriptors are reused.
hadoop.fs.Path was used to relativize locations compared to base
table location and to get the "path" part of the URI. These can
be done with simple String operations if we can assume that the
URIs are normalized.
Results on 1M file 25K partition Iceberg table:
Full load: 13s->10s
Incremental load (0 files): 9s->3.5s
hadoop.fs.Path constructor still uses significant CPU time after
the change, but mainly in functions that run in parallel, so
its effect is not longer that visible in total execution time.
See Jira for before/after flamegraphs.
Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
---
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
2 files changed, 47 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/24052/4
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21817/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Feb 2026 17:52:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Mar 2026 09:07:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24052
to look at the new patch set (#6).
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Quick and dirty solution to speed up IcebergFileMetadataLoader.
Its correctness is based on the assumption that Iceberg file
locations must be normalized.
Noticed in flamegraphs that org.apache.hadoop.fs.Path constructor
is one of the main CPU consumers during Iceberg table loading,
especially incremental reloads when most file descriptors are reused.
hadoop.fs.Path was used to relativize locations compared to base
table location and to get the "path" part of the URI. These can
be done with simple String operations if we can assume that the
URIs are normalized.
Results on 1M file 25K partition Iceberg table:
Full load: 13s->10s
Incremental load (0 files): 9s->3.5s
hadoop.fs.Path constructor still uses significant CPU time after
the change, but mainly in functions that run in parallel, so
its effect is not longer that visible in total execution time.
See Jira for before/after flamegraphs.
Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
---
M fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
3 files changed, 65 insertions(+), 25 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/24052/6
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/24052/5/fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java
File fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java:
http://gerrit.cloudera.org:8080/#/c/24052/5/fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java@228
PS5, Line 228: ;
> nit: double ;;
Done
http://gerrit.cloudera.org:8080/#/c/24052/5/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/24052/5/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@358
PS5, Line 358: Pat
> Path.SEPARATOR?
Done
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Mar 2026 08:51:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21875/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Mar 2026 09:18:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 5:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/24052/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24052/4//COMMIT_MSG@11
PS4, Line 11: locations must be normalized.
> Do we see any signs of this ever changing? I assume it'd be pretty visible
I don't think that writing non-normalized path will be ever ok. Relative paths will be supported at some point, but those should be normalized too.
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
File fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java:
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@a194
PS4, Line 194:
> I wonder if switching to getRawPath() would have been almost as fast. I don
The problem is not with getting the path, but with constructing the Path/Uri. Path uses a few regexps in the constructor, which seemed the biggest CU user.
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@193
PS4, Line 193:
> BaseTable.location() is not a simple getter, it should be moved outside the
Done
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@365
PS4, Line 365:
> We could log the URI here. Maybe with trace-level only, in case there are l
Done
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@355
PS4, Line 355: tring toLookupPath(String path) throws Tabl
> nit: path.length() > tablePathStr_.length()
Done
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@356
PS4, Line 356: if (path.startsWith(tablePathStr_)
> Could tablePathStr_ ever have '/' at the end of it?
It shouldn't have '/' at the end - actually IcebergContentFileStore relies on this when creating absolute paths, as getAbsolutePath(String rootPath) adds an extra / to root path.
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 13:42:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 5: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/24052/5/fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java
File fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java:
http://gerrit.cloudera.org:8080/#/c/24052/5/fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java@228
PS5, Line 228: ;;
nit: double ;;
http://gerrit.cloudera.org:8080/#/c/24052/5/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/24052/5/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@358
PS5, Line 358: '/'
Path.SEPARATOR?
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 14:15:36 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24052
to look at the new patch set (#2).
Change subject: POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
......................................................................
POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
Quick and dirty solution to speed up IcebergFileMetadataLoader,
I am not sure if it is correct.
When working with a ~1M file Iceberg table, noticed that
incremental loading that loads 0 files (e.g. altering a table
property) takes nearly as much time as a full table load. Based
on some jstacks hadoop.fs.Path functions were identified as the
main CPU consumers. Tried replacing them with quick variants
that assume that the parsed URL as correct and fully qualified.
Note sure what can we expect from paths in Iceberg, but I assume
that their validity should be checked only once at most.
Full load: 13s->10s
Incremental load (0 files): 9s->3.5s
Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
---
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
2 files changed, 80 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/24052/2
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21824/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 16:49:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24052
to look at the new patch set (#3).
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Quick and dirty solution to speed up IcebergFileMetadataLoader.
Its correctness is based on the assumption that Iceberg file
locations must be normalized.
Noticed in flamegraphs that org.apache.hadoop.fs.Path constructor
is one of the main CPU consumers during Iceberg table loading,
especially incremental reloads when most file descriptors are reused.
hadoop.fs.Path was used to relativize locations compared to base
table location and to get the "path" part of the URI. These can
be done with simple String operations if we can assume that the
URIs are normalized.
Results on 1M file 25K partition Iceberg table:
Full load: 13s->10s
Incremental load (0 files): 9s->3.5s
hadoop.fs.Path constructor still uses significant CPU time after
the change, but mainly in functions that run in parallel, so
its effect is not longer that visible in total execution time.
See Jira for before/after flamegraphs.
Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
---
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
2 files changed, 49 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/24052/3
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13163/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 27 Feb 2026 17:32:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 4:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/24052/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24052/4//COMMIT_MSG@11
PS4, Line 11: locations must be normalized.
Do we see any signs of this ever changing? I assume it'd be pretty visible if it does?
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
File fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java:
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java@a194
PS4, Line 194:
I wonder if switching to getRawPath() would have been almost as fast. I don't see it doing much different otherwise.
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@356
PS4, Line 356: && path.charAt(tablePathStr_.length()) == '/') {
Could tablePathStr_ ever have '/' at the end of it?
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Mar 2026 16:42:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: POC IMPALA-14792: Try avoiding Hadoop Path functions during Iceberg table loading
......................................................................
Patch Set 2:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/24052/2/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/24052/2/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@367
PS2, Line 367: if (contentFilePathStr.startsWith(tablePathStr_) && contentFilePathStr.charAt(tablePathStr_.length()) == '/') {
line too long (115 > 90)
http://gerrit.cloudera.org:8080/#/c/24052/2/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@378
PS2, Line 378: throw new TableLoadingException(String.format("Failed to load Iceberg datafile " +
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 28 Feb 2026 16:23:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 5: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/24052/4/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@356
PS4, Line 356: if (path.startsWith(tablePathStr_)
> It shouldn't have '/' at the end - actually IcebergContentFileStore relies
Ack
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Mar 2026 16:59:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/24052 )
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
Patch Set 7: Verified+1
The failed test is TestRangerIndependent.test_allow_metadata_update_local_catalog, which should be unrelated. Mergint the change.
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Mar 2026 14:00:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Zoltan Borok-Nagy, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24052
to look at the new patch set (#5).
Change subject: IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
......................................................................
IMPALA-14792: Try avoiding hadoop.fs.Path when loading Iceberg tables
Quick and dirty solution to speed up IcebergFileMetadataLoader.
Its correctness is based on the assumption that Iceberg file
locations must be normalized.
Noticed in flamegraphs that org.apache.hadoop.fs.Path constructor
is one of the main CPU consumers during Iceberg table loading,
especially incremental reloads when most file descriptors are reused.
hadoop.fs.Path was used to relativize locations compared to base
table location and to get the "path" part of the URI. These can
be done with simple String operations if we can assume that the
URIs are normalized.
Results on 1M file 25K partition Iceberg table:
Full load: 13s->10s
Incremental load (0 files): 9s->3.5s
hadoop.fs.Path constructor still uses significant CPU time after
the change, but mainly in functions that run in parallel, so
its effect is not longer that visible in total execution time.
See Jira for before/after flamegraphs.
Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
---
M fe/src/main/java/org/apache/impala/catalog/FileDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/IcebergContentFileStore.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
3 files changed, 65 insertions(+), 25 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/24052/5
--
To view, visit http://gerrit.cloudera.org:8080/24052
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idce89117195e0fa64fdd6a6c576bce09ec2e75ea
Gerrit-Change-Number: 24052
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>