You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Kostas papageorgopoulos (JIRA)" <ji...@apache.org> on 2015/11/16 17:43:10 UTC

[jira] [Comment Edited] (SPARK-11700) Memory leak at SparkContext jobProgressListener stageIdToData map

    [ https://issues.apache.org/jira/browse/SPARK-11700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006895#comment-15006895 ] 

Kostas papageorgopoulos edited comment on SPARK-11700 at 11/16/15 4:42 PM:
---------------------------------------------------------------------------

One workaround to minimize the effect is to Keep the JavaSparkContext forever alive. (Never stop it inside a JVM process that is long running ) and configure the following options {code} 
OPTION                         DEFAULT DESCRIPTION  
spark.ui.retainedJobs	1000	How many jobs the Spark UI and status APIs remember before garbage collecting.
spark.ui.retainedStages 1000 How many stages the Spark UI and status APIs remember before garbage collecting.
spark.worker.ui.retainedExecutors	1000	How many finished executors the Spark UI and status APIs remember before garbage collecting.
spark.worker.ui.retainedDrivers	1000	How many finished drivers the Spark UI and status APIs remember before garbage collecting.
spark.sql.ui.retainedExecutions	1000	How many finished executions the Spark UI and status APIs remember before garbage collecting.
spark.streaming.ui.retainedBatches	1000	How many finished batches the Spark UI and status APIs remember before garbage collecting.{code}  to very small numbers  in order to have the {code}JobProgressListener{code} relevant maps cleaned. 


was (Author: p02096):
One workaround to minimize the effect is to Keep the JavaSparkContext forever alive. (Never stop it inside a JVM process that is long running ) and configure the following options {code} 
spark.ui.retainedJobs	1000	How many jobs the Spark UI and status APIs remember before garbage collecting.
spark.ui.retainedStages 1000 How many stages the Spark UI and status APIs remember before garbage collecting.
spark.worker.ui.retainedExecutors	1000	How many finished executors the Spark UI and status APIs remember before garbage collecting.
spark.worker.ui.retainedDrivers	1000	How many finished drivers the Spark UI and status APIs remember before garbage collecting.
spark.sql.ui.retainedExecutions	1000	How many finished executions the Spark UI and status APIs remember before garbage collecting.
spark.streaming.ui.retainedBatches	1000	How many finished batches the Spark UI and status APIs remember before garbage collecting.{code}  to very small numbers  in order to have the {code}JobProgressListener{code} relevant maps cleaned. 

> Memory leak at SparkContext jobProgressListener stageIdToData map
> -----------------------------------------------------------------
>
>                 Key: SPARK-11700
>                 URL: https://issues.apache.org/jira/browse/SPARK-11700
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core, SQL
>    Affects Versions: 1.5.0, 1.5.1, 1.5.2
>         Environment: Ubuntu 14.04 LTS, Oracle JDK 1.8.51 Apache tomcat 8.0.28. Spring 4
>            Reporter: Kostas papageorgopoulos
>            Priority: Minor
>              Labels: leak, memory-leak
>         Attachments: AbstractSparkJobRunner.java, SparkContextPossibleMemoryLeakIDEA_DEBUG.png, SparkHeapSpaceProgress.png, SparkMemoryAfterLotsOfConsecutiveRuns.png, SparkMemoryLeakAfterLotsOfRunsWithinTheSameContext.png
>
>
> it seems that there is  A SparkContext jobProgressListener memory leak.*. Bellow i describe the  steps i do to reproduce that. 
> I have created a java webapp trying to abstractly Run some Spark Sql jobs that read data from HDFS (join them) and Write them To ElasticSearch using ES hadoop connector. After a Lot of consecutive runs  i noticed that my heap space was full so i got an out of heap space error.
> At the attached file {code} AbstractSparkJobRunner {code} the {code}  public final void run(T jobConfiguration, ExecutionLog executionLog) throws Exception  {code} runs each time an Spark Sql Job is triggered.  So tried to reuse the same SparkContext for a number of consecutive runs. If some rules apply i try to clean up the SparkContext by first calling {code} killSparkAndSqlContext {code}. This code eventually runs {code}  synchronized (sparkContextThreadLock) {
>             if (javaSparkContext != null) {
>                 LOGGER.info("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! CLEARING SPARK CONTEXT!!!!!!!!!!!!!!!!!!!!!!!!!!!");
>                 javaSparkContext.stop();
>                 javaSparkContext = null;
>                 sqlContext = null;
>                 System.gc();
>             }
>             numberOfRunningJobsForSparkContext.getAndSet(0);
>         }
> {code}.
> So at some point in time i suppose that if no other SparkSql job should run i should kill the sparkContext  (The AbstractSparkJobRunner.killSparkAndSqlContext  runs) and this should be garbage collected from garbage collector. However this is not the case, Even if in my debugger shows that my JavaSparkContext object is null see attached picture {code} SparkContextPossibleMemoryLeakIDEA_DEBUG.png {code}.
> The jvisual vm shows an incremental heap space even when the garbage collector is called. See attached picture {code} SparkHeapSpaceProgress.png {code}.
> The memory analyser Tool shows that a big part of the retained heap to be assigned to _jobProgressListener see attached picture {code} SparkMemoryAfterLotsOfConsecutiveRuns.png {code}  and summary picture {code} SparkMemoryLeakAfterLotsOfRunsWithinTheSameContext.png {code}. Although at the same time in Singleton Service the JavaSparkContext is null.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org