You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Matt Cheah (JIRA)" <ji...@apache.org> on 2015/07/07 20:14:04 UTC
[jira] [Commented] (SPARK-7917) Spark doesn't clean up Application
Directories (local dirs)
[ https://issues.apache.org/jira/browse/SPARK-7917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617115#comment-14617115 ]
Matt Cheah commented on SPARK-7917:
-----------------------------------
[~sowen] was there a patch specifically written in master or 1.4.x that fixed this? Can you link a specific PR?
> Spark doesn't clean up Application Directories (local dirs)
> ------------------------------------------------------------
>
> Key: SPARK-7917
> URL: https://issues.apache.org/jira/browse/SPARK-7917
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.0
> Reporter: Zach Fry
> Priority: Minor
>
> Similar to SPARK-4834.
> Spark does clean up the cache and lock files in the local dirs, however, it doesn't clean up the actual directories.
> We have to write custom scripts to go back through the local dirs and find directories that don't contain any files and clear those out.
> Its a pretty simple repro:
> Run a job that does some shuffling, wait for the shuffle files to get cleaned up, go and look on disk at spark.local.dir and notice that the directory(s) are still there, but there are no files in them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org