You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "simhadri-g (via GitHub)" <gi...@apache.org> on 2023/08/01 08:52:47 UTC
[GitHub] [iceberg] simhadri-g opened a new issue, #8198: Request to add KLL Datasketch and hive ColumnStatisticsObj and as standard blob types to puffin file.
simhadri-g opened a new issue, #8198:
URL: https://github.com/apache/iceberg/issues/8198
### Feature Request / Improvement
Hi Everyone,
I have some exciting news to share! Hive now supports writing column statistics to puffin files.
The statistics calculated by Hive include histograms, NDV (Number of Distinct Values), Min and Max values, the number of nulls, the number of true values, column name, and column type. You can find the full list of supported stats here: [Link to GitHub](https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L854C17-L854C30).
These statistics are stored as a Hive columnStatsObject, which is serialized and saved as a blob in puffin. You can refer to the code here for more information: [Link to GitHub](https://github.com/apache/hive/blob/master/iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java#L475).
Currently, this object is supported by Hive and partially by Impala as well: [Link to GitHub](https://github.com/apache/impala/blob/ee069687fcaa06c29404e2220ff577767d905a98/fe/src/main/java/org/apache/impala/catalog/Table.java#L461). We also plan to incorporate the KLL datasketch for histograms.
As a result, we are looking to add columnStatsObject and KLL datasketch as standard blob types for the puffin file. [Link to GitHub](https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/puffin/StandardBlobTypes.java)
Any feedback would be greatly appreciated.
Thanks!
### Query engine
Hive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Request to add KLL Datasketch and hive ColumnStatisticsObj and as standard blob types to puffin file. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8198:
URL: https://github.com/apache/iceberg/issues/8198#issuecomment-2372611269
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Request to add KLL Datasketch and hive ColumnStatisticsObj and as standard blob types to puffin file. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #8198:
URL: https://github.com/apache/iceberg/issues/8198#issuecomment-2339369316
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Request to add KLL Datasketch and hive ColumnStatisticsObj and as standard blob types to puffin file. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #8198: Request to add KLL Datasketch and hive ColumnStatisticsObj and as standard blob types to puffin file.
URL: https://github.com/apache/iceberg/issues/8198
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org