[jira] [Commented] (HIVE-9234) HiveServer2 leaks FileSystem objects in FileSystem.CACHE

["Thejas M Nair (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-9234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14300915#comment-14300915 ] 

Thejas M Nair commented on HIVE-9234:
-------------------------------------

The leak was in FileSystem.CACHE, it would keep growing. The Key in the CACHE includes UGI object. To clear the entries in CACHE corresponding to the UGI object, closeAllForUGI should be called once the UGI is no longer going to be used. 
The fix is in changing the sequence - to do ShimLoader.getHadoopShims().closeAllForUGI(sessionUgi) as the final step of HiveSessionImplwithUGI.close, after super.close().  super.close() was indirectly adding another entry into the CACHE.


> HiveServer2 leaks FileSystem objects in FileSystem.CACHE
> --------------------------------------------------------
>
>                 Key: HIVE-9234
>                 URL: https://issues.apache.org/jira/browse/HIVE-9234
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 0.12.0, 0.13.0, 0.12.1, 0.14.0, 0.13.1
>            Reporter: Vaibhav Gumashta
>            Assignee: Vaibhav Gumashta
>             Fix For: 0.15.0, 1.0.0
>
>         Attachments: HIVE-9234.1.patch, HIVE-9234.2.patch, HIVE-9234.2.patch, HIVE-9234.branch-14.patch
>
>
> Running over extended period (48+ hrs), we've noticed HiveServer2 leaking FileSystem objects in FileSystem.CACHE. Linked jiras were previous attempts to fix it, but the issue still seems to be there. A workaround is to disable the caching (by setting {{fs.hdfs.impl.disable.cache}} and {{fs.file.impl.disable.cache}} to {{true}}), but creating new FileSystem objects is expensive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)