You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ma...@apache.org on 2013/12/24 06:09:26 UTC
[9/9] git commit: Merge pull request #277 from tdas/scheduler-update
Merge pull request #277 from tdas/scheduler-update
Refactored the streaming scheduler and added StreamingListener interface
- Refactored the streaming scheduler for cleaner code. Specifically, the JobManager was renamed to JobScheduler, as it does the actual scheduling of Spark jobs to the SparkContext. The earlier Scheduler was renamed to JobGenerator, as it actually generates the jobs from the DStreams. The JobScheduler starts the JobGenerator. Also, moved all the scheduler related code from spark.streaming to spark.streaming.scheduler package.
- Implemented the StreamingListener interface, similar to SparkListener. The streaming version of StatusReportListener prints the batch processing time statistics (for now). Added StreamingListernerSuite to test it.
- Refactored streaming TestSuiteBase for deduping code in the other streaming testsuites.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/23a9ae6b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/23a9ae6b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/23a9ae6b
Branch: refs/heads/master
Commit: 23a9ae6be33ab2721005b766615660e07eec739f
Parents: 11107c9 6eaa050
Author: Matei Zaharia <ma...@databricks.com>
Authored: Tue Dec 24 00:08:48 2013 -0500
Committer: Matei Zaharia <ma...@databricks.com>
Committed: Tue Dec 24 00:08:48 2013 -0500
----------------------------------------------------------------------
.../apache/spark/scheduler/SparkListener.scala | 7 +-
.../spark/scheduler/SparkListenerBus.scala | 1 -
.../org/apache/spark/streaming/Checkpoint.scala | 2 +-
.../org/apache/spark/streaming/DStream.scala | 11 +-
.../apache/spark/streaming/DStreamGraph.scala | 1 +
.../scala/org/apache/spark/streaming/Job.scala | 41 -----
.../org/apache/spark/streaming/JobManager.scala | 88 ----------
.../spark/streaming/NetworkInputTracker.scala | 174 ------------------
.../org/apache/spark/streaming/Scheduler.scala | 131 --------------
.../spark/streaming/StreamingContext.scala | 26 +--
.../api/java/JavaStreamingContext.scala | 8 +
.../streaming/dstream/ForEachDStream.scala | 3 +-
.../streaming/dstream/NetworkInputDStream.scala | 1 +
.../spark/streaming/scheduler/BatchInfo.scala | 55 ++++++
.../apache/spark/streaming/scheduler/Job.scala | 41 +++++
.../streaming/scheduler/JobGenerator.scala | 131 ++++++++++++++
.../streaming/scheduler/JobScheduler.scala | 108 ++++++++++++
.../spark/streaming/scheduler/JobSet.scala | 68 +++++++
.../scheduler/NetworkInputTracker.scala | 175 +++++++++++++++++++
.../streaming/scheduler/StreamingListener.scala | 75 ++++++++
.../scheduler/StreamingListenerBus.scala | 81 +++++++++
.../spark/streaming/BasicOperationsSuite.scala | 12 --
.../spark/streaming/CheckpointSuite.scala | 26 ++-
.../apache/spark/streaming/FailureSuite.scala | 13 +-
.../spark/streaming/InputStreamsSuite.scala | 12 --
.../streaming/StreamingListenerSuite.scala | 71 ++++++++
.../apache/spark/streaming/TestSuiteBase.scala | 32 +++-
.../spark/streaming/WindowOperationsSuite.scala | 14 +-
28 files changed, 889 insertions(+), 519 deletions(-)
----------------------------------------------------------------------