[jira] [Commented] (SENTRY-705) Possible memory leak of HiveAuthzBinding.authzBindingMap

["Binglin Chang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SENTRY-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536756#comment-14536756 ] 

Binglin Chang commented on SENTRY-705:
--------------------------------------

Add some logging and found the root cause. HiveAuthzBinding.set is called twice when compile(each with different queryId), but in execute, clear can only remove the last added object in authzBindingMap. so each successful query will end up add a HiveAuthzBinding object to authzBindingMap, never got freed.
{noformat}
2015-05-10 00:13:49,414 WARN [HiveServer2-Handler-Pool: Thread-28]: authz.HiveAuthzBinding (HiveAuthzBinding.java:set(157)) - authzBindingMap.set 3acea9e4-a715-4890-b051-b473f242a1ff_3
java.lang.Exception
at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.set(HiveAuthzBinding.java:157)
at org.apache.sentry.binding.hive.HiveAuthzBindingHook.authorizeWithHiveBindings(HiveAuthzBindingHook.java:527)
at org.apache.sentry.binding.hive.HiveAuthzBindingHook.postAnalyze(HiveAuthzBindingHook.java:326)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:438)
......
2015-05-10 00:13:49,415 WARN [HiveServer2-Handler-Pool: Thread-28]: authz.HiveAuthzBinding (HiveAuthzBinding.java:set(157)) - authzBindingMap.set 3acea9e4-a715-4890-b051-b473f242a1ff_4
java.lang.Exception
at org.apache.sentry.binding.hive.authz.HiveAuthzBinding.set(HiveAuthzBinding.java:157)
at org.apache.sentry.binding.hive.HiveAuthzBindingHook.postAnalyze(HiveAuthzBindingHook.java:343)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:438)
......
2015-05-10 00:13:49,424 INFO [HiveServer2-Background-Pool: Thread-50]: ql.Driver (Driver.java:execute(1206)) - Starting command: select 1+1
2015-05-10 00:13:49,424 WARN [HiveServer2-Background-Pool: Thread-50]: authz.HiveAuthzBinding (HiveAuthzBinding.java:clear(169)) - authzBindingMap.remove 3acea9e4-a715-4890-b051-b473f242a1ff_4
{noformat}

> Possible memory leak of HiveAuthzBinding.authzBindingMap
> --------------------------------------------------------
>
>                 Key: SENTRY-705
>                 URL: https://issues.apache.org/jira/browse/SENTRY-705
>             Project: Sentry
>          Issue Type: Bug
>    Affects Versions: 1.4.0
>            Reporter: Binglin Chang
>
> We are using hive0.13 with sentry, version cdh5.2.0, noticed that hiveserver uses lot of memory and doesn't get freed.
> Memory profiling showed that there are lots(millions) of HiveConf and HiveAuthzConf instances hold by HiveAuthzBinding.authzBindingMap
> Look at the code, related call order:
> Driver.compile
>    HiveAuthzBindingHook.preAnalyze
>    HiveAuthzBindingHook.postAnalyze
>       hiveAuthzBinding.set
> Driver.execute
>    HiveAuthzBindingPreExecHook.run
>       hiveAuthzBinding.clear
> Between Driver.compile and Driver.execute, there are too many logic(which sentry can not control), not sure hiveAuthzBinding.clear is guaranteed to be called



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)