You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/01/13 21:18:26 UTC
[9/9] git commit: Merge pull request #400 from tdas/dstream-move
Merge pull request #400 from tdas/dstream-move
Moved DStream and PairDSream to org.apache.spark.streaming.dstream
Similar to the package location of `org.apache.spark.rdd.RDD`, `DStream` has been moved from `org.apache.spark.streaming.DStream` to `org.apache.spark.streaming.dstream.DStream`. I know that the package name is a little long, but I think its better to keep it consistent with Spark's structure.
Also fixed persistence of windowed DStream. The RDDs generated generated by windowed DStream are essentially unions of underlying RDDs, and persistent these union RDDs would store numerous copies of the underlying data. Instead setting the persistence level on the windowed DStream is made to set the persistence level of the underlying DStream.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/b93f9d42
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/b93f9d42
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/b93f9d42
Branch: refs/heads/master
Commit: b93f9d42f21f03163734ef97b2871db945e166da
Parents: e6ed13f ffa1d38
Author: Patrick Wendell <pw...@gmail.com>
Authored: Mon Jan 13 12:18:05 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Mon Jan 13 12:18:05 2014 -0800
----------------------------------------------------------------------
docs/streaming-programming-guide.md | 2 +-
.../examples/RecoverableNetworkWordCount.scala | 2 +-
.../spark/streaming/flume/FlumeUtils.scala | 4 +-
.../spark/streaming/kafka/KafkaUtils.scala | 6 +-
.../apache/spark/streaming/mqtt/MQTTUtils.scala | 4 +-
.../spark/streaming/twitter/TwitterUtils.scala | 7 +-
.../spark/streaming/zeromq/ZeroMQUtils.scala | 3 +-
.../org/apache/spark/streaming/DStream.scala | 756 ------------------
.../spark/streaming/DStreamCheckpointData.scala | 128 ----
.../apache/spark/streaming/DStreamGraph.scala | 6 +-
.../spark/streaming/PairDStreamFunctions.scala | 621 ---------------
.../spark/streaming/StreamingContext.scala | 5 +-
.../spark/streaming/api/java/JavaDStream.scala | 3 +-
.../streaming/api/java/JavaDStreamLike.scala | 1 +
.../streaming/api/java/JavaPairDStream.scala | 1 +
.../api/java/JavaStreamingContext.scala | 5 +-
.../spark/streaming/dstream/DStream.scala | 757 +++++++++++++++++++
.../dstream/DStreamCheckpointData.scala | 126 +++
.../streaming/dstream/FileInputDStream.scala | 2 +-
.../streaming/dstream/FilteredDStream.scala | 2 +-
.../dstream/FlatMapValuedDStream.scala | 2 +-
.../streaming/dstream/FlatMappedDStream.scala | 2 +-
.../streaming/dstream/ForEachDStream.scala | 2 +-
.../streaming/dstream/GlommedDStream.scala | 2 +-
.../spark/streaming/dstream/InputDStream.scala | 2 +-
.../dstream/MapPartitionedDStream.scala | 2 +-
.../streaming/dstream/MapValuedDStream.scala | 2 +-
.../spark/streaming/dstream/MappedDStream.scala | 2 +-
.../dstream/PairDStreamFunctions.scala | 622 +++++++++++++++
.../dstream/ReducedWindowedDStream.scala | 2 +-
.../streaming/dstream/ShuffledDStream.scala | 2 +-
.../spark/streaming/dstream/StateDStream.scala | 2 +-
.../streaming/dstream/TransformedDStream.scala | 2 +-
.../spark/streaming/dstream/UnionDStream.scala | 3 +-
.../streaming/dstream/WindowedDStream.scala | 17 +-
.../streaming/util/MasterFailureTest.scala | 2 +-
.../spark/streaming/BasicOperationsSuite.scala | 1 +
.../spark/streaming/CheckpointSuite.scala | 2 +-
.../spark/streaming/StreamingContextSuite.scala | 3 +-
.../streaming/StreamingListenerSuite.scala | 1 +
.../apache/spark/streaming/TestSuiteBase.scala | 2 +-
.../spark/streaming/WindowOperationsSuite.scala | 15 +
.../tools/JavaAPICompletenessChecker.scala | 50 +-
43 files changed, 1613 insertions(+), 1570 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b93f9d42/streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b93f9d42/streaming/src/main/scala/org/apache/spark/streaming/dstream/StateDStream.scala
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b93f9d42/streaming/src/main/scala/org/apache/spark/streaming/util/MasterFailureTest.scala
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b93f9d42/streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/b93f9d42/streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala
----------------------------------------------------------------------