You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/01/21 07:26:00 UTC
[2/2] git commit: Merge pull request #482 from
tdas/streaming-example-fix
Merge pull request #482 from tdas/streaming-example-fix
Added StreamingContext.awaitTermination to streaming examples
StreamingContext.start() currently starts a non-daemon thread which prevents termination of a Spark Streaming program even if main function has exited. Since the expected behavior of a streaming program is to run until explicitly killed, this was sort of fine when spark streaming applications are launched from the command line. However, when launched in Yarn-standalone mode, this did not work as the driver effectively got terminated when the main function exits. So SparkStreaming examples did not work on Yarn.
This addition to the examples ensures that the examples work on Yarn and also ensures that everyone learns that StreamingContext.awaitTermination() being necessary for SparkStreaming programs to wait.
The true bug-fix of making sure all threads by Spark Streaming are daemon threads is left for post-0.9.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/0367981d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/0367981d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/0367981d
Branch: refs/heads/master
Commit: 0367981d47761cdccd8a44fc6fe803079979c5e3
Parents: 7373ffb 2e95174
Author: Patrick Wendell <pw...@gmail.com>
Authored: Mon Jan 20 22:25:50 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Mon Jan 20 22:25:50 2014 -0800
----------------------------------------------------------------------
.../org/apache/spark/streaming/examples/JavaFlumeEventCount.java | 1 +
.../org/apache/spark/streaming/examples/JavaKafkaWordCount.java | 1 +
.../org/apache/spark/streaming/examples/JavaNetworkWordCount.java | 1 +
.../java/org/apache/spark/streaming/examples/JavaQueueStream.java | 1 +
.../scala/org/apache/spark/streaming/examples/ActorWordCount.scala | 1 +
.../scala/org/apache/spark/streaming/examples/FlumeEventCount.scala | 1 +
.../scala/org/apache/spark/streaming/examples/HdfsWordCount.scala | 1 +
.../scala/org/apache/spark/streaming/examples/KafkaWordCount.scala | 1 +
.../scala/org/apache/spark/streaming/examples/MQTTWordCount.scala | 1 +
.../org/apache/spark/streaming/examples/NetworkWordCount.scala | 1 +
.../scala/org/apache/spark/streaming/examples/RawNetworkGrep.scala | 1 +
.../spark/streaming/examples/RecoverableNetworkWordCount.scala | 1 +
.../apache/spark/streaming/examples/StatefulNetworkWordCount.scala | 1 +
.../org/apache/spark/streaming/examples/TwitterAlgebirdCMS.scala | 1 +
.../org/apache/spark/streaming/examples/TwitterAlgebirdHLL.scala | 1 +
.../org/apache/spark/streaming/examples/TwitterPopularTags.scala | 1 +
.../scala/org/apache/spark/streaming/examples/ZeroMQWordCount.scala | 1 +
17 files changed, 17 insertions(+)
----------------------------------------------------------------------