You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Jeongdae Kim (Jira)" <ji...@apache.org> on 2021/08/23 08:15:00 UTC
[jira] [Work started] (HIVE-25472) Prevent hive-server2 from
getting OOM(Compressed space size) (Backport HIVE-18920)
[ https://issues.apache.org/jira/browse/HIVE-25472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HIVE-25472 started by Jeongdae Kim.
-------------------------------------------
> Prevent hive-server2 from getting OOM(Compressed space size) (Backport HIVE-18920)
> ----------------------------------------------------------------------------------
>
> Key: HIVE-25472
> URL: https://issues.apache.org/jira/browse/HIVE-25472
> Project: Hive
> Issue Type: Bug
> Affects Versions: 2.3.8
> Reporter: Jeongdae Kim
> Assignee: Jeongdae Kim
> Priority: Major
> Attachments: image-2021-08-11-22-03-07-523.jpg, screenshot-5.png, screenshot-6.png
>
>
> Our hive servers are getting shutdown regularly by OOM.
> {code:java}
> Terminating due to java.lang.OutOfMemoryError: Compressed class space {code}
>
> We found out a lot of classes generated by janino compiler exist from heap dump,
> (about 98% of all classes loaded)
> !screenshot-5.png|width=418,height=280!
>
> , and those generated classes are cached (in JaninoRelMetadataProvider)
> !screenshot-6.png|width=424,height=594!
>
> This cache has no expiration, and hive server makes new metadata providers, one of keys for caching, every query, which means hive servers make metadata classes generated in runtime every query and we can't utilize the cache, and finally those classes can't be loaded due to lack of meta space.
>
> By this issue, hive servers are getting slow down because it takes too much time to load classes, until OOM, as below flame graph.
> (48% of sampling is class loading)
> !image-2021-08-11-22-03-07-523.jpg|width=405,height=209!
>
> I think we can fix this issue by either
> a) maintain a static metadata provider (HIVE-18920)
> or
> b) add an option to make constant sizing cache (https://issues.apache.org/jira/browse/CALCITE-1808)
>
> To apply b), we need to upgrade calcite version to 1.15, but this includes lots of changes.
> it may be inappropriate for patch releases.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)