You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Mihaly Szjatinya (Code Review)" <ge...@cloudera.org> on 2026/10/13 14:57:47 UTC
[Impala-ASF-CR] IMPALA-15004: [WIP] Write NDV stats to Iceberg Puffin files in COMPUTE STATS
Mihaly Szjatinya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/24986 )
Change subject: IMPALA-15004: [WIP] Write NDV stats to Iceberg Puffin files in COMPUTE STATS
......................................................................
IMPALA-15004: [WIP] Write NDV stats to Iceberg Puffin files in COMPUTE STATS
COMPUTE STATS on an Iceberg table previously stored column NDV
statistics only in the Hive Metastore (HMS). This change additionally
writes them as an Iceberg Puffin statistics file using the standard
apache-datasketches-theta-v1 blob type, so the NDVs live with the table
metadata and interoperate with other engines such as Trino and Spark.
Impala's existing Puffin read path already consumes such files, so the
written stats are picked up on the next metadata load.
For an Iceberg table, COMPUTE STATS now issues a third child query that
computes a compact theta sketch per column with ds_theta_sketch_iceberg,
which encodes values per Iceberg Appendix D single-value serialization.
The coordinator forwards the serialized sketches to catalogd in
TAlterTableUpdateStatsParams; catalogd writes them to a Puffin file with
the Iceberg Java API and commits it via UpdateStatistics against the
snapshot the stats were computed for.
Testing:
- tests/query_test/test_iceberg.py::
TestIcebergTableWithPuffinStats.test_compute_stats_writes_puffin:
read->write Puffin cycle within Impala
- tests/custom_cluster/test_iceberg_trino_puffin_sketches.py:
Impala<->Trino interop over Puffin theta sketches
Regression:
- tests/metadata/test_compute_stats.py
- tests/query_test/test_iceberg.py::TestIcebergTable::test_compute_stats
- tests/query_test/test_iceberg.py::TestIcebergTableWithPuffinStats
- tests/query_test/test_datasketches.py::TestDatasketches::test_theta
Change-Id: I0112492a9fb156b5317d8e0a6bb3dcf18ae6eeec
Assisted-by: Claude Opus 4.8 (GitHub Copilot)
---
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/client-request-state.cc
M common/thrift/JniCatalog.thrift
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A fe/src/main/java/org/apache/impala/service/IcebergPuffinStatsWriter.java
A testdata/workloads/functional-query/queries/QueryTest/iceberg-trino-interop-puffin-stats.test
A tests/custom_cluster/test_iceberg_trino_puffin_sketches.py
M tests/query_test/test_iceberg.py
10 files changed, 705 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/24986/2
--
To view, visit http://gerrit.cloudera.org:8080/24986
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0112492a9fb156b5317d8e0a6bb3dcf18ae6eeec
Gerrit-Change-Number: 24986
Gerrit-PatchSet: 2
Gerrit-Owner: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Mihaly Szjatinya <ms...@pm.me>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>