[jira] [Commented] (SPARK-14660) Executors show up active tasks indefinitely after stage is killed

["saurabh paliwal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-14660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247513#comment-15247513 ] 

saurabh paliwal commented on SPARK-14660:
-----------------------------------------

After some debugging, we found that there was an issue in handleTaskCompletion method of DagScheduler class
Previously, it was -

if (!stageIdToStage.contains(task.stageId)) {
      // Skip all the actions if the stage has been cancelled.
}

We changed to -

if (!stageIdToStage.contains(task.stageId)) {
      // Skip all the actions if the stage has been cancelled.
      if(event.reason == Success) {
        val attemptId = task.stageAttemptId
        listenerBus.post(SparkListenerTaskEnd(stageId, attemptId, taskType, event.reason,
          event.taskInfo, event.taskMetrics))
      }
      return
    }

This apparently fixed the issue.
The problem was if some task is finished after the cancelTasks is called, the stageIdToStage map doesn't contain the entry anymore but we need to send taskEnd event anyway.
Please review/comment if it's incorrect.

> Executors show up active tasks indefinitely after stage is killed
> -----------------------------------------------------------------
>
>                 Key: SPARK-14660
>                 URL: https://issues.apache.org/jira/browse/SPARK-14660
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core, YARN
>    Affects Versions: 1.6.1
>         Environment: YARN
>            Reporter: saurabh paliwal
>            Priority: Minor
>
> If a job is running and we kill it from all stages UI page, the executors on which the tasks were running for that job keep showing them as active tasks, and the executor won't get lost even after executorIdleTimeout seconds if you have dynamic allocated executors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org