[jira] [Commented] (HIVE-13531) Cache in json_tuple UDF grows larger than it should

["Jürgen Thomann (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-13531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288935#comment-15288935 ] 

Jürgen Thomann commented on HIVE-13531:
---------------------------------------

I investigated the problem now a bit more after the second heap dump and this problem can be reproduced if this UDF is used at the same time in multiple queries.

I'm not sure which is the best version to solve this problem, but there are at least 2 possible fixes.
1. Change the HashCache to a synchronized Map which is easily done with Collections.synchronizedMap
2. remove the static from the declaration of jsonObjectCache. I not sure why it is static, but if two different queries uses json_tuple they would use the same cache at the moment which would reduce the effective cache size for each query.

Another thing is the use of INIT_SIZE = 32 and CACHE_SIZE = 16 with a load factor of 0.6f. Wouldn't it make more sense to increase the load factor to nearly one and increase the CACHE_SIZE to 28 or something in that area?

> Cache in json_tuple UDF grows larger than it should
> ---------------------------------------------------
>
>                 Key: HIVE-13531
>                 URL: https://issues.apache.org/jira/browse/HIVE-13531
>             Project: Hive
>          Issue Type: Bug
>          Components: UDF
>    Affects Versions: 1.1.0
>         Environment: CDH 5.5.0 with Java 1.8.0_45
>            Reporter: Jürgen Thomann
>            Assignee: Jason Dere
>            Priority: Minor
>
> According to the code in ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFJSONTuple.java the HashCache should never grow larger than 16 entries. In the last OOM of Hive Server 2 I found this HashCache with over 1 million java.util.LinkedHashMap$Entry objects.
> The code looks right and works single threaded as it should when I tested it isolated. The only problem I can imagine with my limited Hive source code knowledge that it is accessed concurrently and somewhere the cleanup with removeEldestEntry is not working in that case.
> I had this problem with Hive 1.1.0 but the current implementation in master looks the same for the HashCache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)