You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/01/14 09:05:50 UTC
[5/5] git commit: Merge pull request #416 from tdas/filestream-fix
Merge pull request #416 from tdas/filestream-fix
Removed unnecessary DStream operations and updated docs
Removed StreamingContext.registerInputStream and registerOutputStream - they were useless. InputDStream has been made to register itself, and just registering a DStream as output stream cause RDD objects to be created but the RDDs will not be computed at all.. Also made DStream.register() private[streaming] for the same reasons.
Updated docs, specially added package documentation for streaming package.
Also, changed NetworkWordCount's input storage level to use MEMORY_ONLY, replication on the local machine causes warning messages (as replication fails) which is scary for a new user trying out his/her first example.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/980250b1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/980250b1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/980250b1
Branch: refs/heads/master
Commit: 980250b1ee0cdba9cf06ea87c790a2d504bbf03e
Parents: 055be5c f8bd828
Author: Patrick Wendell <pw...@gmail.com>
Authored: Tue Jan 14 00:05:37 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Tue Jan 14 00:05:37 2014 -0800
----------------------------------------------------------------------
docs/streaming-programming-guide.md | 32 ++++++++--------
.../streaming/examples/NetworkWordCount.scala | 3 +-
.../spark/streaming/flume/FlumeUtils.scala | 1 -
.../streaming/flume/FlumeStreamSuite.scala | 2 +-
.../spark/streaming/kafka/KafkaUtils.scala | 4 +-
.../apache/spark/streaming/mqtt/MQTTUtils.scala | 4 +-
.../spark/streaming/twitter/TwitterUtils.scala | 4 +-
.../spark/streaming/StreamingContext.scala | 40 +++-----------------
.../api/java/JavaStreamingContext.scala | 7 ----
.../spark/streaming/dstream/DStream.scala | 20 +++++-----
.../spark/streaming/dstream/InputDStream.scala | 24 +++++++-----
.../streaming/dstream/NetworkInputDStream.scala | 9 +++--
.../dstream/PairDStreamFunctions.scala | 5 +++
.../org/apache/spark/streaming/package.scala | 38 +++++++++++++++++++
.../streaming/util/MasterFailureTest.scala | 2 +-
.../apache/spark/streaming/JavaTestUtils.scala | 3 +-
.../spark/streaming/BasicOperationsSuite.scala | 1 -
.../spark/streaming/CheckpointSuite.scala | 2 +-
.../spark/streaming/InputStreamsSuite.scala | 8 ++--
.../spark/streaming/StreamingContextSuite.scala | 1 -
.../apache/spark/streaming/TestSuiteBase.scala | 7 +---
21 files changed, 110 insertions(+), 107 deletions(-)
----------------------------------------------------------------------