You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Lantao Jin (Jira)" <ji...@apache.org> on 2019/11/28 11:48:00 UTC
[jira] [Created] (SPARK-30069) Clean up non-shuffle disk block
manager files following executor exists on YARN
Lantao Jin created SPARK-30069:
----------------------------------
Summary: Clean up non-shuffle disk block manager files following executor exists on YARN
Key: SPARK-30069
URL: https://issues.apache.org/jira/browse/SPARK-30069
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 3.0.0
Reporter: Lantao Jin
Currently we only clean up the local directories on application removed. However, when executors die and restart repeatedly, many temp files are left untouched in the local directories, which is undesired behavior and could cause disk space used up gradually.
SPARK-24340 had fixed this problem in standalone mode. But in YARN mode, this issue still exists. Especially, in long running service like Spark thrift-server with dynamic resource allocation disabled, it's very easy causes local disk full.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org