[jira] [Updated] (OOZIE-1683) UserGroupInformationService should close any filesystems opened by it

["Robert Kanter (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OOZIE-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Kanter updated OOZIE-1683:
---------------------------------

    Attachment: OOZIE-1683.patch

The patch closes the FileSystems opened by any UGIs that the UserGroupInformationService has on shutdown.  It also removes some redundant code.  

No tests, but I verified that the FileSystems are now being closed and removed from the HDFS FileSystem cache.  

> UserGroupInformationService should close any filesystems opened by it
> ---------------------------------------------------------------------
>
>                 Key: OOZIE-1683
>                 URL: https://issues.apache.org/jira/browse/OOZIE-1683
>             Project: Oozie
>          Issue Type: Bug
>    Affects Versions: trunk
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: OOZIE-1683.patch
>
>
> When you create a FileSystem object in HDFS, it caches it.  However, when using a UGI to get the FileSystem, it caches it to that object, not its “value”.  In other words, if you create two UGIs for foo to impersonate bar, then they will each create an entry in the cache.  In Oozie, we created a UserGroupInformationService class that basically sits on top of the HDFS cache and caches the UGI objects for impersonating a user so we can reuse the same FileSystem object each time.  However, when running unit tests, we typically destroy and recreate the services for each test (sometimes even more) so Oozie’s cache was being lost and we were effectively creating a new FileSystem object all the time without closing it. 
> Looking at a heap dump after running many of the tests, we saw that there were 800+ FileSystem objects open for the same FileSystem.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)