[jira] [Commented] (HADOOP-13971) Fix memory leak in FileSystem.Cache.Key class

["Shilun Fan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17803031#comment-17803031 ] 

Shilun Fan commented on HADOOP-13971:
-------------------------------------

Bulk update: moved all 3.4.0 non-blocker issues, please move back if it is a blocker. Retarget 3.5.0.

> Fix memory leak in FileSystem.Cache.Key class
> ---------------------------------------------
>
>                 Key: HADOOP-13971
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13971
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, security
>            Reporter: Wei-Chiu Chuang
>            Priority: Major
>
> Calling {{FileSystem#get(final URI uri, final Configuration conf, final String user)}} multiple times can result in memory leak because of the hash method implementation of UserGroupInformation. FileSystem always instantiates a new FileSystem object despite using the same user name/same URI.
> In the past, other downstream projects work around this bug by either disabling cache (set {{fs.%s.impl.disable.cache}} to {{true}}) or call {{FileSystem.closeAllForUGI()}} to release resource on demand. (See for instance HIVE-3098, YARN-58, TEZ-1585)
> However, neither approach is desirable. The first workaround loses performance because it disables cache. This bug was discussed extensively in  HADOOP-12707, but the proposed workaround {{FileSystem.closeAllForUGI()}} is insufficient, because it won't purge the objects from cache due to the same hash method implementation bug.
> I would like to file a new jira, knowing that current workarounds do not work, and invite more discussion. An ideal approach is to change UGI hash method, but it may break many downstream applications, so setting target version as 3.0.0-beta



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org