You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Csaba Ringhofer (Jira)" <ji...@apache.org> on 2026/02/27 14:16:00 UTC
[jira] [Created] (IMPALA-14792) Incremental updates of Iceberg tables is slow even with 0 new files
Csaba Ringhofer created IMPALA-14792:
----------------------------------------
Summary: Incremental updates of Iceberg tables is slow even with 0 new files
Key: IMPALA-14792
URL: https://issues.apache.org/jira/browse/IMPALA-14792
Project: IMPALA
Issue Type: Improvement
Components: Catalog
Reporter: Csaba Ringhofer
Noticed with a very big Iceberg table (25K partitions, ~1M files) that incremental refresh is not much faster than full load.
Full reload: 13s
{code}
0227 11:35:53.533893 2966519 IcebergFileMetadataLoader.java:311] 194b9270d0ad58d9:d277d65700000000] Collected 956989 Iceberg content files into 25000 partitions. Duration: 2s747ms
I20260227 11:35:53.533991 2966519 ParallelFileMetadataLoader.java:230] 194b9270d0ad58d9:d277d65700000000] Parallel Iceberg file metadata listing using a thread pool of size 5
I20260227 11:36:03.145077 2966519 IcebergTable.java:548] 194b9270d0ad58d9:d277d65700000000] Loaded file and block metadata for default.bigice. Time taken: 13s166ms
{code}
reload after ALTER TABLE SET TBLPROPERTY: 9s
{code}
I20260227 11:25:19.225029 2964808 HdfsTable.java:1311] 50478fc5b1d7a62f:4609a73e00000000] Incrementally loaded table metadata for: default.bigice
I20260227 11:25:22.142279 2964808 IcebergFileMetadataLoader.java:311] 50478fc5b1d7a62f:4609a73e00000000] Collected 0 Iceberg content files into 0 partitions. Duration: 21.044us
I20260227 11:25:26.278229 2964808 IcebergTable.java:548] 50478fc5b1d7a62f:4609a73e00000000] Loaded file and block metadata for default.bigice. Time taken: 8s835msI20260227
{code}
Based on some random jstacks most time is spent dealing with pathes:
{code}
java.lang.Thread.State: RUNNABLE
at java.net.URI$Parser.scan(java.base@17.0.17/URI.java:3082)
at java.net.URI$Parser.parseAuthority(java.base@17.0.17/URI.java:3261)
at java.net.URI$Parser.parseHierarchical(java.base@17.0.17/URI.java:3221)
at java.net.URI$Parser.parse(java.base@17.0.17/URI.java:3177)
at java.net.URI.<init>(java.base@17.0.17/URI.java:781)
at org.apache.hadoop.fs.Path.initialize(Path.java:259)
at org.apache.hadoop.fs.Path.<init>(Path.java:220)
at org.apache.impala.catalog.IcebergFileMetadataLoader.getOldFd(IcebergFileMetadataLoader.java:359)
at org.apache.impala.catalog.IcebergFileMetadataLoader.loadContentFilesWithOldFds(IcebergFileMetadataLoader.java:188)
at org.apache.impala.catalog.IcebergFileMetadataLoader.loadInternal(IcebergFileMetadataLoader.java:130)
at org.apache.impala.catalog.IcebergFileMetadataLoader.load(IcebergFileMetadataLoader.java:98)
at org.apache.impala.catalog.IcebergTable.loadFileMetadata(IcebergTable.java:534)
at org.apache.impala.catalog.IcebergTable.load(IcebergTable.java:467)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)