You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@kudu.apache.org by "HeLifu (JIRA)" <ji...@apache.org> on 2019/05/16 10:18:00 UTC

[jira] [Commented] (KUDU-2366) LockManager consumes significant memory

    [ https://issues.apache.org/jira/browse/KUDU-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841185#comment-16841185 ] 

HeLifu commented on KUDU-2366:
------------------------------

Maybe we can refer to redis's operation for dict rehash which could expand and shrink. :)

> LockManager consumes significant memory
> ---------------------------------------
>
>                 Key: KUDU-2366
>                 URL: https://issues.apache.org/jira/browse/KUDU-2366
>             Project: Kudu
>          Issue Type: Improvement
>          Components: tablet
>    Affects Versions: 1.7.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Major
>
> Looking at a heap dump of a server that's been running for a while with an ingest workload across multiple tables, I see the LockManager is using about 200MB of RAM. The workload in this case has batches of about 30,000 rows each, so while each batch is in flight the LockManager hashtable has that many locks in it. That causes it to expand to the next higher power of two (64k slots). Each slot takes 16 bytes, so the lock table is reaching about 1MB. We never resize _down_, so even once the tablet becomes cold, it still uses 1M of unrecoverable RAM for the rest of the tserver lifetime.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)