You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2019/09/23 11:48:00 UTC
[jira] [Resolved] (SPARK-29177) Zombie tasks prevents executor from
releasing when task exceeds maxResultSize
[ https://issues.apache.org/jira/browse/SPARK-29177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-29177.
---------------------------------
Fix Version/s: 3.0.0
2.4.5
Resolution: Fixed
Issue resolved by pull request 25850
[https://github.com/apache/spark/pull/25850]
> Zombie tasks prevents executor from releasing when task exceeds maxResultSize
> -----------------------------------------------------------------------------
>
> Key: SPARK-29177
> URL: https://issues.apache.org/jira/browse/SPARK-29177
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.3.4, 2.4.4
> Reporter: Adrian Wang
> Assignee: Adrian Wang
> Priority: Major
> Fix For: 2.4.5, 3.0.0
>
>
> When we fetch results from executors and found the total size has exceeded the maxResultSize configured, Spark will simply abort the stage and all dependent jobs. But the task triggered this is actually successful, but never post out `TaskEnd` event, as a result it will never be removed from `CoarseGrainedSchedulerBackend`. If dynamic allocation is enabled, there will be zombie executor(s) remaining in resource manager, it will never die until application ends.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org