[jira] [Commented] (IMPALA-15004) Puffin stats writer for Iceberg tables

["Mihaly Szjatinya (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-15004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118287#comment-18118287 ] 

Mihaly Szjatinya commented on IMPALA-15004:
-------------------------------------------

In the parent epic IMPALA-14879 there is also this option:

> 3. keep storing missing stats in HMS, but improve them to provide incremental support (e.g. store snapshot id)

In the scope of this issue it would mean only write Theta sketches for NDV to Puffin. This gives 'Iceberg-native' incremental stats and interop with other engines. 

As for the other stats, make them incremental on HMS, at least for now. That doesn't seem as much effort.

> Puffin stats writer for Iceberg tables
> --------------------------------------
>
>                 Key: IMPALA-15004
>                 URL: https://issues.apache.org/jira/browse/IMPALA-15004
>             Project: IMPALA
>          Issue Type: New Feature
>            Reporter: Zoltán Borók-Nagy
>            Assignee: Mihaly Szjatinya
>            Priority: Major
>              Labels: impala-iceberg, impala-iceberg-active-backlog
>
> Currently COMPUTE STATS store column statistics only in HMS.
> Iceberg has Puffin files for this purpose, but currently there's only a single blob type (Apache Theta sketches) we can store that only supports NDV.
> Impala should comply to Iceberg's standards and write Puffin files. The stats that cannot be stored in well-known Iceberg Puffin blob types could be stored in custom Impala blobs. That way all statistics information could be retrieved from a single place.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org