You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Krisztian Kasa (Jira)" <ji...@apache.org> on 2026/09/03 13:57:00 UTC
[jira] [Commented] (HIVE-29541) Imprecise NDV stats on Iceberg partition columns
[ https://issues.apache.org/jira/browse/HIVE-29541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111118#comment-18111118 ]
Krisztian Kasa commented on HIVE-29541:
---------------------------------------
In case of iceberg tables the Describe statement aggregates the partition stats comes from the puffin file via the storage handler. The partition level column stats are aggregate by {{HiveIcebergStorageHandler.getAggrColStatsFor}}
https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java#L892-L896
It is using {{HyperLogLog}} for merging nvd values of partition level ndvs.
https://github.com/apache/hive/blob/bff3870c637184d096c1d6789761e0c0afdd6037/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/columnstats/aggr/LongColumnStatsAggregator.java#L107
> Imprecise NDV stats on Iceberg partition columns
> ------------------------------------------------
>
> Key: HIVE-29541
> URL: https://issues.apache.org/jira/browse/HIVE-29541
> Project: Hive
> Issue Type: Bug
> Components: Statistics
> Reporter: Stamatis Zampetakis
> Priority: Major
> Attachments: col_stats_part_ndv.q
>
>
> The number of distinct values (NDV/countDistinct) statistic is slightly off for Iceberg partition columns. Currently, the NDV stats for Iceberg (partition and regular) columns is computed by aggregating the individual stats from each partition. The aggregation logic is subject to a small margin of error since there is no way to have a fully accurate result from the moment that we rely on probabilistic data structures (i.e., HyperLogLog).
> However, for partition columns we know exactly how many partitions are present in the table so we don't need to rely on probabilistic data structures since the NDV is equal to the number of partitions (no complex aggregation needed). The StatsUtils class already contains some logic (getColStatsForPartCol) to compute the NDV along with some other stats directly from partitions but this does not kick in for Iceberg tables.
> The problem can be seen also in qtests after loading the LINEITEM table from TPC_0_001 database in an Iceberg table using the L_ORDERKEY as a partition key and running DESCRIBE FORMATTED on the partitioning column.
> {code:sql}
> DESC FORMATTED ice.lineitem l_orderkey
> {code}
> {noformat}
> col_name L_ORDERKEY
> data_type int
> min 1
> max 5988
> num_nulls 0
> distinct_count 1523
> avg_col_len
> max_col_len
> num_trues
> num_falses
> bit_vector HL
> comment
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"l_comment\":\"true\",\"l_commitdate\":\"true\",\"l_discount\":\"true\",\"l_extendedprice\":\"true\",\"l_linenumber\":\"true\",\"l_linestatus\":\"true\",\"l_orderkey\":\"true\",\"l_partkey\":\"true\",\"l_quantity\":\"true\",\"l_receiptdate\":\"true\",\"l_returnflag\":\"true\",\"l_shipdate\":\"true\",\"l_shipinstruct\":\"true\",\"l_shipmode\":\"true\",\"l_suppkey\":\"true\",\"l_tax\":\"true\"}}
> {noformat}
> Observe that distinct_count (NDV) is 1523 while the real number is 1500. In non-Iceberg tables the NDV is accurate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)