You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-issues@hadoop.apache.org by "Wei-Chiu Chuang (Jira)" <ji...@apache.org> on 2020/02/21 01:14:00 UTC

[jira] [Created] (HADOOP-16876) KMS delegation tokens are memory expensive

Wei-Chiu Chuang created HADOOP-16876:
----------------------------------------

             Summary: KMS delegation tokens are memory expensive
                 Key: HADOOP-16876
                 URL: https://issues.apache.org/jira/browse/HADOOP-16876
             Project: Hadoop Common
          Issue Type: Bug
          Components: kms
            Reporter: Wei-Chiu Chuang
         Attachments: Screen Shot 2020-02-20 at 5.04.12 PM.png

We recently saw a number of users reporting memory consumption in KMS.

Part of the reason being HADOOP-14445. Without that, the number of kms delegation tokens that zookeeper stores is proportional to the number of KMS servers.

There are two problems:
(1) it exceeds zookeeper jute buffer length and operations fail.
(2) KMS use more heap memory to store KMS DTs.

But even with HADOOP-14445, KMS DTs are still expensive. Looking at a heap dump from KMS, the majority of the heap is occupied by znode and KMS DT objects. With the growing number of encrypted clusters and use cases, this is increasingly a problem our users encounter.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org