[jira] [Created] (SPARK-11700) Possible memory leak at SparkContext jobProgressListener stageIdToData map

["Kostas papageorgopoulos (JIRA)" <ji...@apache.org>]

Kostas papageorgopoulos created SPARK-11700:
-----------------------------------------------

             Summary: Possible memory leak at SparkContext jobProgressListener stageIdToData map
                 Key: SPARK-11700
                 URL: https://issues.apache.org/jira/browse/SPARK-11700
             Project: Spark
          Issue Type: Question
          Components: Spark Core, SQL
    Affects Versions: 1.5.1, 1.5.0, 1.5.2
         Environment: Ubuntu 14.04 LTS, Oracle JDK 1.8.51 Apache tomcat 8.0.28. Spring 4
            Reporter: Kostas papageorgopoulos
            Priority: Minor


I have created a java webapp trying to abstractly Run some Spark Sql jobs that read data from HDFS (join them) and Write them To ElasticSearch using ES hadoop connector. After a Lot of consecutive runs  i noticed that my heap space was full so i got an out of heap space error.

At the attached file {code} AbstractSparkJobRunner {code} the {code}  public final void run(T jobConfiguration, ExecutionLog executionLog) throws Exception  {code} runs each time an Spark Sql Job is triggered.  So tried to reuse the same SparkContext for a number of consecutive runs. If some rules apply i try to clean up the SparkContext by first calling {code} killSparkAndSqlContext {code}. This code eventually runs {code}  synchronized (sparkContextThreadLock) {
            if (javaSparkContext != null) {
                LOGGER.info("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! CLEARING SPARK CONTEXT!!!!!!!!!!!!!!!!!!!!!!!!!!!");
                javaSparkContext.stop();
                javaSparkContext = null;
                sqlContext = null;

                System.gc();
            }
            numberOfRunningJobsForSparkContext.getAndSet(0);
        }
{code}.

So at some point in time i suppose that if no other SparkSql job should run i should kill the sparkContext and this should be garbage collected from garbage collector. However this is not the case, Even if in my debugger see attached picture {code} SparkContextPossibleMemoryLeakIDEA_DEBUG.png {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org