You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Taeyun Kim (JIRA)" <ji...@apache.org> on 2015/05/07 10:18:59 UTC
[jira] [Created] (SPARK-7439) Should delete temporary local
directories
Taeyun Kim created SPARK-7439:
---------------------------------
Summary: Should delete temporary local directories
Key: SPARK-7439
URL: https://issues.apache.org/jira/browse/SPARK-7439
Project: Spark
Issue Type: Bug
Components: Block Manager
Affects Versions: 1.3.1
Environment: Windows 7, CentOS 6.6
Reporter: Taeyun Kim
Priority: Minor
Spark does not delete temporary local directories.
After a spark program completes, there are 3 temporary directories remain in the temp directory. The directory names are like this: spark-2e389487-40cc-4a82-a5c7-353c0feefbb7
The directories are empty.
They are created every time the Spark program runs. So the number of files and directories keeps growing.
I've traced the spark source code.
The module methods that create the 3 'temp' directories are as follows:
* DiskBlockManager.createLocalDirs
* HttpFileServer.initialize
* SparkEnv.sparkFilesDir
They (eventually) call Utils.getOrCreateLocalRootDirs and then Utils.createDirectory, which intentionally does NOT mark the directory for automatic deletion.
The comment of createDirectory method says: "The directory is guaranteed to be newly created, and is not marked for automatic deletion."
But since the directories does not hold useful data after the program completes, they should be deleted if possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org