[jira] [Commented] (HIVE-9017) Clean up temp files of RSC [Spark Branch]

["Marcelo Vanzin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-9017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14244944#comment-14244944 ] 

Marcelo Vanzin commented on HIVE-9017:
--------------------------------------

These files are created by Spark when downloading resources for the app (e.g. application jars). In standalone mode, by default, these files will end up in /tmp (java.io.tmpdir). The problem is that the app doesn't clean up these files; in fact, it can't, because they are supposed to be shared in case multiple executors run on the same host - so one executor cannot unilaterally decide to delete them.

(That's not entirely true; I guess it could, but then it would cause other executors to re-download the file when needed, so more overhead.)

This is not a problem in Yarn mode, since the temp dir is under a Yarn-managed directory that is deleted when the app shuts down.

So, while I think of a clean way to fix this in Spark, the following can be done on the Hive side:

- create an app-specific temp directory before launching the Spark app
- set {{spark.local.dir}} to that location
- delete the directory when the client shuts down

> Clean up temp files of RSC [Spark Branch]
> -----------------------------------------
>
>                 Key: HIVE-9017
>                 URL: https://issues.apache.org/jira/browse/HIVE-9017
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Rui Li
>
> Currently RSC will leave a lot of temp files in {{/tmp}}, including {{*_lock}}, {{*_cache}}, {{spark-submit.*.properties}}, etc.
> We should clean up these files or it will exhaust disk space.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)