[jira] [Commented] (HDFS-12748) NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY

["Weiwei Yang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226534#comment-16226534 ] 

Weiwei Yang commented on HDFS-12748:
------------------------------------

Hi [~yangjiandan]

Thanks for reporting this issue, the leak must be fixed. I took a look at {{NamenodeWebHdfsMethods}}, there are currently 2 places calling FileSystem.get() without closing. Can we fix them both?

bq. which contains 7,844,890 DistributedFileSystem

First impression I was thinking the {{FileSystem#Cache}} is not working properly, but after looking into the code, I think it is working as expected. The cache stores FileSystem instances distinguished by {{FileSystem.Cache.Key}} which includes a UGI field. According to HADOOP-6670, UGI is distinguished by the {{Subject}}. In webhdfs, UGI is parsed by {{DataNodeUGIProvider}} per http request, this way {{Subject}} is an different instance every time so it will be new to the cache. That was why the cache has so many instances if not properly closing.

> NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY
> ------------------------------------------------------------
>
>                 Key: HDFS-12748
>                 URL: https://issues.apache.org/jira/browse/HDFS-12748
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>    Affects Versions: 2.8.2
>            Reporter: Jiandan Yang 
>
> In our production environment, the standby NN often do fullgc, through mat we found the largest object is FileSystem$Cache, which contains 7,844,890 DistributedFileSystem.
> By view hierarchy of method FileSystem.get() , I found only NamenodeWebHdfsMethods#get call FileSystem.get(). I don't know why creating different DistributedFileSystem every time instead of get a FileSystem from cache.
> {code:java}
>     case GETHOMEDIRECTORY: {
>       final String js = JsonUtil.toJsonString("Path",
>           FileSystem.get(conf != null ? conf : new Configuration())
>               .getHomeDirectory().toUri().getPath());
>       return Response.ok(js).type(MediaType.APPLICATION_JSON).build();
>     }
> {code}
> When we close FileSystem when GETHOMEDIRECTORY, NN don't do fullgc.
> {code:java}
>     case GETHOMEDIRECTORY: {
>       FileSystem fs = null;
>       try {
>         fs = FileSystem.get(conf != null ? conf : new Configuration());
>         final String js = JsonUtil.toJsonString("Path",
>             fs.getHomeDirectory().toUri().getPath());
>         return Response.ok(js).type(MediaType.APPLICATION_JSON).build();
>       } finally {
>         if (fs != null) {
>           fs.close();
>         }
>       }
>     }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org