[jira] [Commented] (HIVE-11408) HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used

["Sergey Shelukhin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-11408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694002#comment-14694002 ] 

Sergey Shelukhin commented on HIVE-11408:
-----------------------------------------

that seems a little bit risky. Maybe not use Hadoop reflectionutils instead? :)
In llap branch we made our own ReflectionUtil that also doesn't have the epic lock.

> HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used
> -------------------------------------------------------------------------------
>
>                 Key: HIVE-11408
>                 URL: https://issues.apache.org/jira/browse/HIVE-11408
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 0.13.0, 0.14.0, 0.13.1, 1.0.0, 1.1.1
>            Reporter: Vaibhav Gumashta
>            Assignee: Vaibhav Gumashta
>         Attachments: HIVE-11408.1.patch
>
>
> I'm able to reproduce with 0.14. I'm yet to see if HIVE-10453 fixes the issue (since it's on top of a larger patch: HIVE-2573 that was added in 1.2). 
> Basically, add jar creates a new classloader for loading the classes from the new jar and adds the new classloader to the SessionState object of user's session, making the older one its parent. Creating a temporary function uses the new classloader to load the class used for the function. On closing a session, although there is code to close the classloader for the session, I'm not seeing the new classloader getting GCed and from the heapdump I can see it holds on to the temporary function's class that should have gone away after the session close. 
> Steps to reproduce:
> 1.
> {code}
> jdbc:hive2://localhost:10000/> add jar hdfs:///tmp/audf.jar;
> {code}
> 2. 
> Use a profiler (I'm using yourkit) to verify that a new URLClassLoader was added.
> 3. 
> {code}
> jdbc:hive2://localhost:10000/> CREATE TEMPORARY FUNCTION funcA AS 'org.gumashta.udf.AUDF'; 
> {code}
> 4. 
> Close the jdbc session.
> 5. 
> Take the memory snapshot and verify that the new URLClassLoader is indeed there and is holding onto the class it loaded (org.gumashta.udf.AUDF) for the session which we already closed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)