[jira] [Commented] (SPARK-1865) Improve behavior of cleanup of disk state

["Mridul Muralidharan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000716#comment-14000716 ] 

Mridul Muralidharan commented on SPARK-1865:
--------------------------------------------

You could also modify the startup script for executors to do this ... So that we don't rely on spark executors to do housekeeping by always assuming clean exit.
This is what yarn does for example.

> Improve behavior of cleanup of disk state
> -----------------------------------------
>
>                 Key: SPARK-1865
>                 URL: https://issues.apache.org/jira/browse/SPARK-1865
>             Project: Spark
>          Issue Type: Improvement
>          Components: Deploy, Spark Core
>            Reporter: Aaron Davidson
>
> Right now the behavior of disk cleanup is centered around the exit hook of the executor, which attempts to cleanup shuffle files and disk manager blocks, but may fail. We should make this behavior more predictable, perhaps by letting the Standalone Worker cleanup the disk state, and adding a flag to disable having the executor cleanup its own state.



--
This message was sent by Atlassian JIRA
(v6.2#6252)