[jira] [Assigned] (IMPALA-12298) Improve incremental load of Iceberg tables

["Zoltán Borók-Nagy (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-12298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zoltán Borók-Nagy reassigned IMPALA-12298:
------------------------------------------

    Assignee: Zoltán Borók-Nagy

> Improve incremental load of Iceberg tables
> ------------------------------------------
>
>                 Key: IMPALA-12298
>                 URL: https://issues.apache.org/jira/browse/IMPALA-12298
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Catalog
>            Reporter: Zoltán Borók-Nagy
>            Assignee: Zoltán Borók-Nagy
>            Priority: Major
>              Labels: impala-iceberg, performance
>
> *The followings mostly affect HDFS/Ozone where we need to contact the NameNode to create file descriptors with block locations. On cloud object stores where there are no block locations, we only need the Iceberg metadata to create the file descriptors.*
> Currently we always reload all the metadata belonging to an Iceberg table.
> This means we recreate all the file descriptors even if only a few of them have changed.
> We could check the amount of the newly added files, and if there's only a few of them then we should only load the file descriptors for those one by one.
> We can fallback to a full reload if a significant amount of files have changed, i.e. when it is better to use a recursive file listing.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org