You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Charles Allen (JIRA)" <ji...@apache.org> on 2015/12/22 19:51:46 UTC
[jira] [Commented] (SPARK-1865) Improve behavior of cleanup of disk
state
[ https://issues.apache.org/jira/browse/SPARK-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068542#comment-15068542 ]
Charles Allen commented on SPARK-1865:
--------------------------------------
This is compounded by the fact that some of the shutdown processes will stop() during the normal course of the main thread, then will fail to wait for completion if stop() is ALSO called via the shutdown hook.
> Improve behavior of cleanup of disk state
> -----------------------------------------
>
> Key: SPARK-1865
> URL: https://issues.apache.org/jira/browse/SPARK-1865
> Project: Spark
> Issue Type: Improvement
> Components: Deploy, Spark Core
> Reporter: Aaron Davidson
>
> Right now the behavior of disk cleanup is centered around the exit hook of the executor, which attempts to cleanup shuffle files and disk manager blocks, but may fail. We should make this behavior more predictable, perhaps by letting the Standalone Worker cleanup the disk state, and adding a flag to disable having the executor cleanup its own state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org