You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2013/12/25 01:35:51 UTC
[20/20] git commit: Merge pull request #127 from
kayousterhout/consolidate_schedulers
Merge pull request #127 from kayousterhout/consolidate_schedulers
Deduplicate Local and Cluster schedulers.
The code in LocalScheduler/LocalTaskSetManager was nearly identical
to the code in ClusterScheduler/ClusterTaskSetManager. The redundancy
made making updating the schedulers unnecessarily painful and error-
prone. This commit combines the two into a single TaskScheduler/
TaskSetManager.
Unfortunately the diff makes this change look much more invasive than it is -- TaskScheduler.scala is only superficially changed (names updated, overrides removed) from the old ClusterScheduler.scala, and the same with
TaskSetManager.scala.
Thanks @rxin for suggesting this change!
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/85a344b4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/85a344b4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/85a344b4
Branch: refs/heads/master
Commit: 85a344b4f0cd149c6e6f06f8b942c34146b302be
Parents: c2dd6bc 1efe3ad
Author: Patrick Wendell <pw...@gmail.com>
Authored: Tue Dec 24 16:35:06 2013 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Tue Dec 24 16:35:06 2013 -0800
----------------------------------------------------------------------
.../scala/org/apache/spark/SparkContext.scala | 41 +-
.../spark/scheduler/ExecutorLossReason.scala | 38 +
.../spark/scheduler/SchedulerBackend.scala | 37 +
.../spark/scheduler/TaskResultGetter.scala | 107 +++
.../apache/spark/scheduler/TaskScheduler.scala | 11 +-
.../spark/scheduler/TaskSchedulerImpl.scala | 474 ++++++++++++
.../apache/spark/scheduler/TaskSetManager.scala | 704 +++++++++++++++++-
.../apache/spark/scheduler/WorkerOffer.scala | 24 +
.../scheduler/cluster/ClusterScheduler.scala | 473 ------------
.../cluster/ClusterTaskSetManager.scala | 713 -------------------
.../cluster/CoarseGrainedSchedulerBackend.scala | 6 +-
.../scheduler/cluster/ExecutorLossReason.scala | 38 -
.../scheduler/cluster/SchedulerBackend.scala | 37 -
.../cluster/SimrSchedulerBackend.scala | 4 +-
.../cluster/SparkDeploySchedulerBackend.scala | 6 +-
.../scheduler/cluster/TaskResultGetter.scala | 108 ---
.../spark/scheduler/cluster/WorkerOffer.scala | 24 -
.../mesos/CoarseMesosSchedulerBackend.scala | 5 +-
.../cluster/mesos/MesosSchedulerBackend.scala | 7 +-
.../spark/scheduler/local/LocalBackend.scala | 108 +++
.../spark/scheduler/local/LocalScheduler.scala | 222 ------
.../scheduler/local/LocalTaskSetManager.scala | 191 -----
.../scala/org/apache/spark/FailureSuite.scala | 4 +-
.../SparkContextSchedulerCreationSuite.scala | 58 +-
.../spark/scheduler/ClusterSchedulerSuite.scala | 265 +++++++
.../org/apache/spark/scheduler/FakeTask.scala | 26 +
.../spark/scheduler/SparkListenerSuite.scala | 19 +-
.../spark/scheduler/TaskResultGetterSuite.scala | 112 +++
.../spark/scheduler/TaskSetManagerSuite.scala | 319 +++++++++
.../cluster/ClusterSchedulerSuite.scala | 267 -------
.../cluster/ClusterTaskSetManagerSuite.scala | 319 ---------
.../spark/scheduler/cluster/FakeTask.scala | 27 -
.../cluster/TaskResultGetterSuite.scala | 114 ---
.../scheduler/local/LocalSchedulerSuite.scala | 227 ------
.../cluster/YarnClusterScheduler.scala | 10 +-
35 files changed, 2287 insertions(+), 2858 deletions(-)
----------------------------------------------------------------------