You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by nitin <ni...@gmail.com> on 2015/02/24 16:46:54 UTC

Does Spark delete shuffle files of lost executor in running system(on YARN)?

Hi All,

I noticed that Spark doesn't delete local shuffle files of a lost executor
in a running system(running in yarn-client mode). For long running system,
this might fill up disk space in case of frequent executor failures. Can we
delete these files when executor loss reported to driver?

Thanks
-Nitin



--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Does-Spark-delete-shuffle-files-of-lost-executor-in-running-system-on-YARN-tp10755.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org