You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:13:15 UTC

[jira] [Resolved] (SPARK-20116) Remove task-level functionality from the DAGScheduler

     [ https://issues.apache.org/jira/browse/SPARK-20116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-20116.
----------------------------------
    Resolution: Incomplete

> Remove task-level functionality from the DAGScheduler
> -----------------------------------------------------
>
>                 Key: SPARK-20116
>                 URL: https://issues.apache.org/jira/browse/SPARK-20116
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Scheduler
>    Affects Versions: 2.2.0
>            Reporter: Kay Ousterhout
>            Assignee: Kay Ousterhout
>            Priority: Major
>              Labels: bulk-closed
>
> Long, long ago, the scheduler code was more modular, and the DAGScheduler handled the logic of scheduling DAGs of stages (as the name suggests) and the TaskSchedulerImpl handled scheduling the tasks within a stage.  Over time, more and more task-specific functionality has been added to the DAGScheduler, and now, the DAGScheduler duplicates a bunch of the task tracking that's done by other scheduler components.  This makes the scheduler code harder to reason about, and has led to some tricky bugs (e.g., SPARK-19263).  We should move all of this functionality back to the TaskSchedulerImpl and TaskSetManager, which should "hide" that complexity from the DAGScheduler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org