[jira] [Commented] (YARN-4570) Nodemanager leaking RawLocalFilesystem instances for user "testing"

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-4570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089999#comment-15089999 ] 

Jason Lowe commented on YARN-4570:
----------------------------------

The "testing" user shows up in the LocalizerTokenIdentifier.  I suspect there's a way where we can create a local filesystem object while processing a localizer heartbeat.  I believe the heartbeat will have a current user of "testing" since that's the user of the token authenticating the localizer connection.  If we don't close all the filesystems for that UGI then we can end up leaking them, since each remote user is a different UGI even if they have the same username.


> Nodemanager leaking RawLocalFilesystem instances for user "testing"
> -------------------------------------------------------------------
>
>                 Key: YARN-4570
>                 URL: https://issues.apache.org/jira/browse/YARN-4570
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.6.0
>            Reporter: Jason Lowe
>
> I recently ran across a NodeManager that was running slowly due to excessive GC.  Digging into the heap I saw that most of the issue was leaked filesystem statistics data objects which has been fixed in HADOOP-12107.  However I also noticed there were many thousands of RawLocalFilesystem objects on the heap, far more than any other FileSystem type.  Sampling a number of them showed that they were for the "testing" user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)