You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2014/11/06 02:18:33 UTC
[jira] [Commented] (SPARK-4236) External shuffle service must
cleanup its shuffle files
[ https://issues.apache.org/jira/browse/SPARK-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199539#comment-14199539 ]
Apache Spark commented on SPARK-4236:
-------------------------------------
User 'aarondav' has created a pull request for this issue:
https://github.com/apache/spark/pull/3126
> External shuffle service must cleanup its shuffle files
> -------------------------------------------------------
>
> Key: SPARK-4236
> URL: https://issues.apache.org/jira/browse/SPARK-4236
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Aaron Davidson
> Priority: Critical
>
> When external shuffle service is enabled, executors no longer cleanup their own shuffle files, as the external server can continue to serve them. The external serer must clean these files up once it knows that they will never be used again (i.e., when the application terminates or when Spark's ContextCleaner requests that they be deleted).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org