You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Jason Fehr (Jira)" <ji...@apache.org> on 2026/03/12 15:13:00 UTC
[jira] [Updated] (IMPALA-13794) Calculate memory usage estimates more precisely for Iceberg tables
[ https://issues.apache.org/jira/browse/IMPALA-13794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Fehr updated IMPALA-13794:
--------------------------------
Description:
Currently we use HdfsTable's memory usage estimates for Iceberg tables, even though they have different characteristics.
We should come up with a calculation that is better suited for Iceberg tables.
The current estimate doesn't take some things into account, e.g.
* IcebergFileDescriptor sizes are larger than plain FileDescriptor sizes, especially when the table is partitioned
* IcebergContentFileStore's data structures overhead
* V3: Deletion Vectors stored in IcebergContentFileStore.dataFileToDV_
* the internal Iceberg BaseTable object
Also need to cross-check the estimation with the Jamm weigher (that we use in our caches).
was:
Currently we use HdfsTable's memory usage estimates for Iceberg tables, even though they have different characteristics.
We should come up with a calculation that is better suited for Iceberg tables.
> Calculate memory usage estimates more precisely for Iceberg tables
> ------------------------------------------------------------------
>
> Key: IMPALA-13794
> URL: https://issues.apache.org/jira/browse/IMPALA-13794
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Zoltán Borók-Nagy
> Assignee: Jason Fehr
> Priority: Major
> Labels: impala-iceberg, ramp-up
>
> Currently we use HdfsTable's memory usage estimates for Iceberg tables, even though they have different characteristics.
> We should come up with a calculation that is better suited for Iceberg tables.
> The current estimate doesn't take some things into account, e.g.
> * IcebergFileDescriptor sizes are larger than plain FileDescriptor sizes, especially when the table is partitioned
> * IcebergContentFileStore's data structures overhead
> * V3: Deletion Vectors stored in IcebergContentFileStore.dataFileToDV_
> * the internal Iceberg BaseTable object
> Also need to cross-check the estimation with the Jamm weigher (that we use in our caches).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org