Remove COMPLETED applications and shuffle data

[sayantini <sa...@gmail.com>]

Hi All,



Please help me with the below 2 issues:



*Environment:*



I am running my spark cluster in stand alone mode.



I am initializing the spark context from inside my tomcat server.



I am setting below properties in environment.sh in $SPARK_HOME/conf
directory



SPARK_MASTER_OPTS=-Dspark.deploy.retainedApplications=1
-Dspark.deploy.retainedDrivers=1

SPARK_WORKER_OPTS=-Dspark.worker.cleanup.enabled=true
-Dspark.worker.cleanup.interval=600 -Dspark.worker.cleanup.appDataTtl=600



SPARK_LOCAL_DIRS=$user.home/tmp



*Issue 1:*



Still in my $SPARK_HOME/work folder, application-folders continue to grow
as and when I restart the tomcat.



I also tried to stop the spark context (sc.stop()) in tomcat’s
contextDestroyed listener but still I am not able to remove the undesired
application folders.



*Issue 2:*

The ‘tmp’ folder is getting filled up with shuffle data and eating my
entire hard disk. Is there any setting to remove shuffle data of ‘FINISHED’
applications.



Thanks in advance,

Sayantini

Re: Remove COMPLETED applications and shuffle data

[Akhil Das <ak...@sigmoidanalytics.com>]

Try these:

- Disable shuffle : spark.shuffle.spill=false (It might end up in OOM)
- Enable log rotation:

sparkConf.set("spark.executor.logs.rolling.strategy", "size")
.set("spark.executor.logs.rolling.size.maxBytes", "1024")
.set("spark.executor.logs.rolling.maxRetainedFiles", "3")

You can also look into spark.cleaner.ttl
<https://spark.apache.org/docs/latest/configuration.html#execution-behavior>

Thanks
Best Regards

On Tue, May 26, 2015 at 12:23 PM, sayantini <sa...@gmail.com>
wrote:

> Hi All,
>
>
>
> Please help me with the below 2 issues:
>
>
>
> *Environment:*
>
>
>
> I am running my spark cluster in stand alone mode.
>
>
>
> I am initializing the spark context from inside my tomcat server.
>
>
>
> I am setting below properties in environment.sh in $SPARK_HOME/conf
> directory
>
>
>
> SPARK_MASTER_OPTS=-Dspark.deploy.retainedApplications=1
> -Dspark.deploy.retainedDrivers=1
>
> SPARK_WORKER_OPTS=-Dspark.worker.cleanup.enabled=true
> -Dspark.worker.cleanup.interval=600 -Dspark.worker.cleanup.appDataTtl=600
>
>
>
> SPARK_LOCAL_DIRS=$user.home/tmp
>
>
>
> *Issue 1:*
>
>
>
> Still in my $SPARK_HOME/work folder, application-folders continue to grow
> as and when I restart the tomcat.
>
>
>
> I also tried to stop the spark context (sc.stop()) in tomcat’s
> contextDestroyed listener but still I am not able to remove the undesired
> application folders.
>
>
>
> *Issue 2:*
>
> The ‘tmp’ folder is getting filled up with shuffle data and eating my
> entire hard disk. Is there any setting to remove shuffle data of ‘FINISHED’
> applications.
>
>
>
> Thanks in advance,
>
> Sayantini
>