[jira] [Created] (IMPALA-13370) Read Puffin stats from metadata.json property if available

["Daniel Becker (Jira)" <ji...@apache.org>]

Daniel Becker created IMPALA-13370:
--------------------------------------

             Summary: Read Puffin stats from metadata.json property if available
                 Key: IMPALA-13370
                 URL: https://issues.apache.org/jira/browse/IMPALA-13370
             Project: IMPALA
          Issue Type: Improvement
          Components: Frontend
            Reporter: Daniel Becker
            Assignee: Daniel Becker


When Trino writes Puffin stats for a column, it includes the NDV as a property in the "statistics" section of the metadata.json file, in addition to the Theta sketch in the Puffin file. When we are only reading the stats and not writing/updating them, it would be enough to read this property if it is present.

An example of the "statistics" section:
{code:java}
"statistics" : [ {
    "snapshot-id" : 1226095104912303892,
    "statistics-path" : "hdfs://localhost:20500/test-warehouse/iceberg_test/hadoop_catalog/ice/iceberg_query_metadata/metadata/20240829_112839_00004_p6sck-7f433a45-607b-4561-89a3-fc4c58ef60d9.stats",
    "file-size-in-bytes" : 306,
    "file-footer-size-in-bytes" : 257,
    "blob-metadata" : [ {
      "type" : "apache-datasketches-theta-v1",
      "snapshot-id" : 1226095104912303892,
      "sequence-number" : 4,
      "fields" : [ 1 ],
      "properties" : {
        "ndv" : "2"
      }
    } ]
  } ]{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org