[jira] [Resolved] (IMPALA-11583) Use Iceberg APIs to update table properties for Iceberg tables

["Quanlong Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Quanlong Huang resolved IMPALA-11583.
-------------------------------------
    Fix Version/s: Impala 4.2.0
       Resolution: Fixed

> Use Iceberg APIs to update table properties for Iceberg tables
> --------------------------------------------------------------
>
>                 Key: IMPALA-11583
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11583
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Catalog
>            Reporter: Zoltán Borók-Nagy
>            Assignee: Zoltán Borók-Nagy
>            Priority: Major
>              Labels: impala-iceberg
>             Fix For: Impala 4.2.0
>
>
> COMPUTE STATS updates table-level stats via alter_table() HMS API. This replaces the whole HMS table, therefore if there are concurrent modifications by another engine, e.g. Hive, it's possible that these modifications are lost.
> This is critical for Iceberg tables, as the 'metadata_location' table property must always point to the latest snapshot. Inadvertently rewriting it during COMPUTE STATS can result in a data loss.
> Table-level stats like 'numRows' and 'totalSize' are already updated by Iceberg during table modifications, i.e. there is no need to update these values for COMPUTE STATS.
> Column stats are not affected as they are updated via a different API call ([updateTableColumnStatistics|https://github.com/apache/impala/blob/4e813b7085c995a7244ef886b00c22e9d93cc80c/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L1638()]), and it doesn't touch the table properties. But updating statistics also require us to update table property "impala.lastComputeStatsTime".  We should update it via Iceberg APIs when HiveCatalog is used:
> https://github.com/apache/impala/blob/4e813b7085c995a7244ef886b00c22e9d93cc80c/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java#L211
> For other catalogs than HiveCatalog we still need to update the table property via HMS API. It should be safe as other catalogs don't depend on HMS table properties.
> Reloading the HMS table before invoking 'alter_table()' can be considered in other cases (non-Iceberg tables as well), to decrease the possibility of losing concurrent table updates.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)