You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Lantao Jin (Jira)" <ji...@apache.org> on 2019/11/29 08:43:00 UTC

[jira] [Updated] (SPARK-30069) Clean up non-shuffle disk block manager files following executor exists on YARN

     [ https://issues.apache.org/jira/browse/SPARK-30069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lantao Jin updated SPARK-30069:
-------------------------------
    Component/s: Spark Core

> Clean up non-shuffle disk block manager files following executor exists on YARN
> -------------------------------------------------------------------------------
>
>                 Key: SPARK-30069
>                 URL: https://issues.apache.org/jira/browse/SPARK-30069
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core, YARN
>    Affects Versions: 3.0.0
>            Reporter: Lantao Jin
>            Priority: Major
>
> Currently we only clean up the local directories on application removed. However, when executors die and restart repeatedly, many temp files are left untouched in the local directories, which is undesired behavior and could cause disk space used up gradually.
> SPARK-24340 had fixed this problem in standalone mode. But in YARN mode, this issue still exists. Especially, in long running service like Spark thrift-server with dynamic resource allocation disabled, it's very easy causes local disk full.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org