[jira] [Commented] (HADOOP-14907) Memory leak in FileSystem cache

["Daryn Sharp (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181044#comment-16181044 ] 

Daryn Sharp commented on HADOOP-14907:
--------------------------------------

You have a leak, but the screenshot isn't showing the leak.  The highlighted strings are both in the same UGI instance.

Screenshot-1 shows a {{HashMap.Node}} from the {{FileSystem.Cache}}.  The {{HashMap.Node.key}} field above it is a {{FileSystem.Cache.Key}} which references a {{UserGroupInformation}}.  The {{HashMap.Node.value}} field shown is a {{DistributedFileSystem}} instance, which references the same {{FileSystem.Cache.Key}}.  You can see from the hashcodes that the ugi  is identical.

The problem is you have ~20k {{Subject}} instances.  Are you repeatedly invoking {{UserGroupInformation.createRemoteUser}}?

> Memory leak in FileSystem cache
> -------------------------------
>
>                 Key: HADOOP-14907
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14907
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: hdfs-client
>    Affects Versions: 2.7.4
>            Reporter: cen yuhai
>         Attachments: screenshot-1.png, screenshot-2.png
>
>
> There is a memory leak in FileSystem cache. It will take a lot of memory.I think the root cause is that the equals function in class Key is not right. You can see in the screenshot-1.png, the same user etl is in different key... And also FileSystem cache should be a LRU cache



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org