You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2015/07/31 09:17:05 UTC
[jira] [Assigned] (SPARK-8873) Support cleaning up shuffle files
when using shuffle service in Mesos
[ https://issues.apache.org/jira/browse/SPARK-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-8873:
-----------------------------------
Assignee: Apache Spark (was: Timothy Chen)
> Support cleaning up shuffle files when using shuffle service in Mesos
> ---------------------------------------------------------------------
>
> Key: SPARK-8873
> URL: https://issues.apache.org/jira/browse/SPARK-8873
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
> Affects Versions: 1.2.0
> Reporter: Timothy Chen
> Assignee: Apache Spark
> Priority: Critical
> Labels: mesos
>
> With dynamic allocation enabled with Mesos, drivers can launch with shuffle data cached in the external shuffle service.
> However, there is no reliable way to let the shuffle service clean up the shuffle data when the driver exits, since it may crash before it notifies the shuffle service and shuffle data will be cached forever.
> We need to implement a reliable way to detect driver termination and clean up shuffle data accordingly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org