You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/01/13 05:04:44 UTC
[8/8] git commit: Merge pull request #394 from tdas/error-handling
Merge pull request #394 from tdas/error-handling
Better error handling in Spark Streaming and more API cleanup
Earlier errors in jobs generated by Spark Streaming (or in the generation of jobs) could not be caught from the main driver thread (i.e. the thread that called StreamingContext.start()) as it would be thrown in different threads. With this change, after `ssc.start`, one can call `ssc.awaitTermination()` which will be block until the ssc is closed, or there is an exception. This makes it easier to debug.
This change also adds ssc.stop(<stop-spark-context>) where you can stop StreamingContext without stopping the SparkContext.
Also fixes the bug that came up with PRs #393 and #381. MetadataCleaner default value has been changed from 3500 to -1 for normal SparkContext and 3600 when creating a StreamingContext. Also, updated StreamingListenerBus with changes similar to SparkListenerBus in #392.
And changed a lot of protected[streaming] to private[streaming].
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/405bfe86
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/405bfe86
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/405bfe86
Branch: refs/heads/master
Commit: 405bfe86ef9c3021358d2ac89192857478861fe0
Parents: 28a6b0c aa2c993
Author: Patrick Wendell <pw...@gmail.com>
Authored: Sun Jan 12 20:04:21 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Sun Jan 12 20:04:21 2014 -0800
----------------------------------------------------------------------
.../org/apache/spark/util/MetadataCleaner.scala | 2 +-
.../spark/streaming/mqtt/MQTTStreamSuite.scala | 2 +-
.../streaming/twitter/TwitterStreamSuite.scala | 2 +-
.../org/apache/spark/streaming/Checkpoint.scala | 2 +-
.../apache/spark/streaming/ContextWaiter.scala | 28 +++
.../org/apache/spark/streaming/DStream.scala | 62 +++---
.../apache/spark/streaming/DStreamGraph.scala | 8 +-
.../spark/streaming/StreamingContext.scala | 112 +++++-----
.../api/java/JavaStreamingContext.scala | 21 +-
.../spark/streaming/dstream/InputDStream.scala | 2 +-
.../streaming/dstream/NetworkInputDStream.scala | 2 +-
.../apache/spark/streaming/scheduler/Job.scala | 9 +-
.../streaming/scheduler/JobGenerator.scala | 81 ++++---
.../streaming/scheduler/JobScheduler.scala | 141 ++++++++----
.../spark/streaming/scheduler/JobSet.scala | 18 +-
.../scheduler/NetworkInputTracker.scala | 40 ++--
.../streaming/scheduler/StreamingListener.scala | 3 +-
.../scheduler/StreamingListenerBus.scala | 21 +-
.../spark/streaming/util/RecurringTimer.scala | 13 +-
.../spark/streaming/BasicOperationsSuite.scala | 6 +-
.../spark/streaming/CheckpointSuite.scala | 22 ++
.../spark/streaming/StreamingContextSuite.scala | 218 +++++++++++++++++++
.../apache/spark/streaming/TestSuiteBase.scala | 7 +-
23 files changed, 591 insertions(+), 231 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/405bfe86/streaming/src/main/scala/org/apache/spark/streaming/DStream.scala
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/405bfe86/streaming/src/test/scala/org/apache/spark/streaming/BasicOperationsSuite.scala
----------------------------------------------------------------------