[jira] [Commented] (HADOOP-16876) KMS delegation tokens are memory expensive

["Bhavik Patel (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-16876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17679681#comment-17679681 ] 

Bhavik Patel commented on HADOOP-16876:
---------------------------------------

I think this is handled in https://issues.apache.org/jira/browse/HADOOP-16828 & https://issues.apache.org/jira/browse/HDFS-15383 

[~weichiu] [~xyao] Can you confirm ?

> KMS delegation tokens are memory expensive
> ------------------------------------------
>
>                 Key: HADOOP-16876
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16876
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: kms
>            Reporter: Wei-Chiu Chuang
>            Priority: Major
>         Attachments: Screen Shot 2020-02-20 at 5.04.12 PM.png
>
>
> We recently saw a number of users reporting high memory consumption in KMS.
> Part of the reason being HADOOP-14445. Without that, the number of kms delegation tokens that zookeeper stores is proportional to the number of KMS servers.
> There are two problems:
> (1) it exceeds zookeeper jute buffer length and operations fail.
> (2) KMS uses more heap memory to store KMS DTs.
> But even with HADOOP-14445, KMS DTs are still expensive. Looking at a heap dump from KMS, the majority of the heap is occupied by znode and KMS DT objects. With the growing number of encrypted clusters and use cases, this is increasingly a problem our users encounter.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org