You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Fu Lili (Jira)" <ji...@apache.org> on 2024/07/24 03:57:00 UTC

[jira] [Created] (IMPALA-13254) Optimizing incremental reload performance of Iceberg tables

Fu Lili created IMPALA-13254:
--------------------------------

             Summary: Optimizing incremental reload performance of Iceberg tables
                 Key: IMPALA-13254
                 URL: https://issues.apache.org/jira/browse/IMPALA-13254
             Project: IMPALA
          Issue Type: Improvement
          Components: Catalog
    Affects Versions: Impala 4.4.0
            Reporter: Fu Lili
            Assignee: Fu Lili


When performing a {{REFRESH}} on an Iceberg table, if the number of changed files exceeds the {{iceberg_reload_new_files_threshold}} configuration (default is 100), a highly inefficient reload operation is triggered.

The main issue with this code lies in the {{IcebergFileMetadataLoader.getFileStatuses}} function. During incremental loading, the {{listWithLocations}} parameter is always set to {{{}false{}}}, resulting in {{fs.getFileStatus}} and {{fs.getFileBlockLocations}} operations being performed on each {{contentFile}} sequentially (if the filesystem supports {{{}StorageIds{}}}).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org