[jira] [Created] (HIVE-10494) hive metastore server can't release its java heap with no work on it

["liqida (JIRA)" <ji...@apache.org>]

liqida created HIVE-10494:
-----------------------------

             Summary: hive metastore server can't release its java heap with no work on it
                 Key: HIVE-10494
                 URL: https://issues.apache.org/jira/browse/HIVE-10494
             Project: Hive
          Issue Type: Bug
          Components: Metastore
    Affects Versions: 0.13.0
         Environment: cloudera cdh 5.2.0
10 nodes
 128G ram, 10T disk, 32core CPU for each node
using impala for data analysis
            Reporter: liqida


I use impala for data analysis

after a long time runing , impala DDL statements need a long time to complete "Planning finished" and "DML Metastore update finished" steps.

Both of them take 50 seconds and more.

I found that HMS java heap has affected it so much .and I restart the hive metastore server , the problem fixed .

The HMS java ops like this :

-XX:+UseParNewGC 
-XX:+UseConcMarkSweepGC 
-XX:-CMSConcurrentMTEnabled
-XX:CMSInitiatingOccupancyFraction=70 
-XX:+CMSParallelRemarkEnabled 
-XX:+UseCMSCompactAtFullCollection 
-XX:CMSFullGCsBeforeCompaction=0 
-XX:SurvivorRatio=1

and the total heap size is 3GB 

after 3 days or less , I found the old genaration is full , and no matter what kind of GC I tried , it never works .

And then , after the whole work is done() , I run " jmap -F -histo PID "
I found this :

Object Histogram:

num       #instances    #bytes  Class description
--------------------------------------------------------------------------
1:              3955457 696160432       com.mysql.jdbc.JDBC4ResultSet
2:              3942714 630834240       com.mysql.jdbc.StatementImpl
3:              4051520 194472960       java.util.HashMap
4:              4714330 150858560       java.util.HashMap$Entry
5:              3990264 63844224        java.util.HashSet
6:              3978657 63658512        java.util.HashMap$KeySet
7:              3955458 63463696        com.mysql.jdbc.Field[]
8:              3964025 63424400        java.util.concurrent.atomic.AtomicBoolean
9:             3961293 63380688        java.lang.Object


I think this is the causation

So, what can I do with this , should I change some configuration or do something to fix this , or HMS has any CACHE ? THANKS 

BTW: Hive version 0.13.0 , I only use impala 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)