You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/09/01 07:10:00 UTC
[jira] [Assigned] (SPARK-36575) Executor lost may cause spark stage
to hang
[ https://issues.apache.org/jira/browse/SPARK-36575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-36575:
------------------------------------
Assignee: (was: Apache Spark)
> Executor lost may cause spark stage to hang
> -------------------------------------------
>
> Key: SPARK-36575
> URL: https://issues.apache.org/jira/browse/SPARK-36575
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 3.1.2
> Reporter: hujiahua
> Priority: Major
>
> When a executor finished a task of some stage, the driver will receive a `StatusUpdate` event to handle it. At the same time the driver found the executor heartbeat timed out, so the dirver also need handle ExecutorLost event simultaneously. There was a race condition issues here, which will make the task never been rescheduled again and the stage hang over.
> The problem is that `TaskResultGetter.enqueueSuccessfulTask` use asynchronous thread to handle successful task, that mean the synchronized lock of `TaskSchedulerImpl` was released prematurely during midway [https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala#L61]. So `TaskSchedulerImpl` may handle executorLost first, then the asynchronous thread will go on to handle successful task. It cause `TaskSetManager.successful` and `TaskSetManager.tasksSuccessful` wrong result, and the task never been queued again.
> related log when the issue produce:
> 21/08/05 02:58:14,784 INFO [dispatcher-event-loop-8] TaskSetManager: Starting task 4004.0 in stage 1328625.0 (TID 347212402, 10.109.89.3, executor 366724, partition 4004, ANY, 7994 bytes)
> 21/08/05 03:00:24,126 ERROR [dispatcher-event-loop-4] TaskSchedulerImpl: Lost executor 366724 on 10.109.89.3: Executor heartbeat timed out after 140830 ms
> 21/08/05 03:00:24,218 WARN [dispatcher-event-loop-4] TaskSetManager: Lost task 4004.0 in stage 1328625.0 (TID 347212402, 10.109.89.3, executor 366724): ExecutorLostFailure (executor 366724 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 140830 ms
> 21/08/05 03:00:24,542 INFO [task-result-getter-2] TaskSetManager: Finished task 4004.0 in stage 1328625.0 (TID 347212402) in 129758 ms on 10.109.89.3 (executor 366724) (3047/5400)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org