[jira] [Resolved] (IMPALA-13609) Store Iceberg snapshot id for COMPUTE STATS

["Daniel Becker (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-13609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Becker resolved IMPALA-13609.
------------------------------------
    Resolution: Implemented

> Store Iceberg snapshot id for COMPUTE STATS
> -------------------------------------------
>
>                 Key: IMPALA-13609
>                 URL: https://issues.apache.org/jira/browse/IMPALA-13609
>             Project: IMPALA
>          Issue Type: Improvement
>            Reporter: Daniel Becker
>            Assignee: Daniel Becker
>            Priority: Major
>
> Currently, when COMPUTE STATS is run from Impala, we set the 'impala.lastComputeStatsTime' table property. Iceberg Puffin stats, on the other hand, store the snapshot id for which stats were calculated. Although it is possible to retrieve the timestamp of a snapshot, comparing these two values is error-prone, e.g. in the following situation
>  * COMPUTE STATS calculation is running on Snapshot N
>  * Snapshot N+1 is committed at time T
>  * COMPUTE STATS finishes and sets 'impala.lastComputeStatsTime' at time T + Delta
>  * Some engine writes Puffin statistics for Snapshot N+1
> After this, HMS stats will appear to be more recent even though it was calculated on Snapshot N, while we have Puffin stats for Snapshot N+1.
> To resolve this, COMPUTE STATS could set a new table property, e.g. 'impala.computeStatsSnapshotId'.
> On the other hand, COMPUTE STATS could be set to calculate stats for only a subset of the columns, and then a different subset in a subsequent run. The recency of the stats will then be different for each column. We could consider storing the snapshot id on a per column basis.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)