You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues-all@impala.apache.org by "bharath v (JIRA)" <ji...@apache.org> on 2018/08/10 19:34:00 UTC

[jira] [Updated] (IMPALA-7424) Improve in-memory representation of incremental stats

     [ https://issues.apache.org/jira/browse/IMPALA-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

bharath v updated IMPALA-7424:
------------------------------
    Issue Type: Sub-task  (was: Improvement)
        Parent: IMPALA-2649

> Improve in-memory representation of incremental stats
> -----------------------------------------------------
>
>                 Key: IMPALA-7424
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7424
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Catalog
>    Affects Versions: Impala 2.13.0, Impala 3.1.0
>            Reporter: bharath v
>            Assignee: bharath v
>            Priority: Major
>
> Incremental stats are stored in the HMS' parameters map as plain Java Strings. This is suboptimal since Java String class internally uses UTF-16 encoding for the underlying bytes. The idea here is to switch to a byte array representation so that we can reduce the memory usage by half (8 bytes).  We can also compress the byte array using gzip compression and lazily decompress them when needed (typically during the incremental stats computation's finalization phase).
> A prototype of this patch on a real-world Catalog dump showed ~54% JVM heap usage reduction (end-to-end) and ~79% reduction in the heap footprint for the incremental stats.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org