You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "bharath v (JIRA)" <ji...@apache.org> on 2018/08/10 19:31:00 UTC
[jira] [Created] (IMPALA-7424) Improve in-memory representation of
incremental stats
bharath v created IMPALA-7424:
---------------------------------
Summary: Improve in-memory representation of incremental stats
Key: IMPALA-7424
URL: https://issues.apache.org/jira/browse/IMPALA-7424
Project: IMPALA
Issue Type: Improvement
Components: Catalog
Affects Versions: Impala 2.13.0, Impala 3.1.0
Reporter: bharath v
Incremental stats are stored in the HMS' parameters map as plain Java Strings. This is suboptimal since Java String class internally uses UTF-16 encoding for the underlying bytes. The idea here is to switch to a byte array representation so that we can reduce the memory usage by half (8 bytes). We can also compress the byte array using gzip compression and lazily decompress them when needed (typically during the incremental stats computation's finalization phase).
A prototype of this patch on a real-world Catalog dump showed ~54% JVM heap usage reduction (end-to-end) and ~79% reduction in the heap footprint for the incremental stats.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org