You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/11/23 03:30:58 UTC
[jira] [Assigned] (SPARK-18553) Executor loss may cause
TaskSetManager to be leaked
[ https://issues.apache.org/jira/browse/SPARK-18553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-18553:
------------------------------------
Assignee: Josh Rosen (was: Apache Spark)
> Executor loss may cause TaskSetManager to be leaked
> ---------------------------------------------------
>
> Key: SPARK-18553
> URL: https://issues.apache.org/jira/browse/SPARK-18553
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 1.6.0, 2.0.0, 2.1.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Blocker
>
> Due to a bug in TaskSchedulerImpl, the complete sudden loss of an executor may cause a TaskSetManager to be leaked, causing ShuffleDependencies and other data structures to be kept alive indefinitely, leading to various types of resource leaks (including shuffle file leaks).
> In a nutshell, the problem is that TaskSchedulerImpl did not maintain its own mapping from executorId to running task ids, leaving it unable to clean up taskId to taskSetManager maps when an executor is totally lost.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org