You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2018/08/17 18:43:00 UTC
[jira] [Resolved] (IMPALA-7406) Flatbuffer wrappers use almost as
much memory as underlying data
[ https://issues.apache.org/jira/browse/IMPALA-7406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon resolved IMPALA-7406.
---------------------------------
Resolution: Fixed
Fix Version/s: Impala 3.1.0
> Flatbuffer wrappers use almost as much memory as underlying data
> ----------------------------------------------------------------
>
> Key: IMPALA-7406
> URL: https://issues.apache.org/jira/browse/IMPALA-7406
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Major
> Fix For: Impala 3.1.0
>
>
> Currently the file descriptors stored in the catalogd memory for each partition use a FlatBuffer to reduce the number of separate objects on the Java heap. However, the FlatBuffer objects internally each store a ByteBuffer and int position, so each object takes 32 bytes on its own. The ByteBuffer takes 56 bytes since it stores various references, endianness, limit, mark, position, etc. This amounts to about 88 bytes overhead on top of the actual underlying flatbuf byte array which is typically around 100 bytes for a single-block file. So, we're have about a 1:1 ratio of memory overhead and a 2:1 ratio of object count overhead for each partition.
> If we simply stored the byte[] array and constructed wrappers on demand, we'd save 88 bytes and 2 objects per partition. The downside is that we'd need to do short-lived ByteBuffer allocations at access time, and based on some benchmarking I did, they don't get escape-analyzed out. So, it's not a super clear win, but still worth considering.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org