You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "zuotingbing (JIRA)" <ji...@apache.org> on 2017/12/15 07:17:00 UTC

[jira] [Created] (SPARK-22793) Memory leak in Spark Thrift Server

zuotingbing created SPARK-22793:
-----------------------------------

             Summary: Memory leak in Spark Thrift Server
                 Key: SPARK-22793
                 URL: https://issues.apache.org/jira/browse/SPARK-22793
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.0.2
            Reporter: zuotingbing
            Priority: Critical


1. Start HiveThriftServer2
2. Connect to thriftserver through beeline
3. Close the beeline
4. repeat step2 and step 3 for several times

we found there are many directories never be dropped under the path
{code:java}
hive.exec.local.scratchdir
{code} and 
{code:java}
hive.exec.scratchdir
{code} , as we know the scratchdir is added to deleteOnExit when it be created. So it means that the cache size of FileSystem deleteOnExit will keep increasing until JVM terminated.

In addition, we use 
{code:java}
jmap -histo:live [PID]
{code} to printout the size of objects in HiveThriftServer2 Process, we can find the object "org.apache.spark.sql.hive.client.HiveClientImpl" and "org.apache.hadoop.hive.ql.session.SessionState" keep increasing even though we closed all the beeline connections, which caused the leak of Memory.






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org