[jira] [Created] (SPARK-7917) Spark doesn't clean up Application Directories (local dirs)

["Zach Fry (JIRA)" <ji...@apache.org>]

Zach Fry created SPARK-7917:
-------------------------------

             Summary: Spark doesn't clean up Application Directories (local dirs) 
                 Key: SPARK-7917
                 URL: https://issues.apache.org/jira/browse/SPARK-7917
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.3.0
            Reporter: Zach Fry


Similar to SPARK-4834. 

Spark does clean up the cache and lock files in the local dirs, however, it doesn't clean up the actual directories. 

We have to write custom scripts to go back through the local dirs and find directories that don't contain any files and clear those out. 

Its a pretty simple repro: 
Run a job that does some shuffling, wait for the shuffle files to get cleaned up, go and look on disk at spark.local.dir and notice that the directory(s) are still there, but there are no files in them.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org