You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by td...@apache.org on 2014/11/19 22:17:23 UTC
spark git commit: [SPARK-4481][Streaming][Doc] Fix the wrong
description of updateFunc
Repository: spark
Updated Branches:
refs/heads/master 22fc4e751 -> 3bf7ceebb
[SPARK-4481][Streaming][Doc] Fix the wrong description of updateFunc
Removed `If `this` function returns None, then corresponding state key-value pair will be eliminated.` for the description of `updateFunc: (Iterator[(K, Seq[V], Option[S])]) => Iterator[(K, S)]`
Author: zsxwing <zs...@gmail.com>
Closes #3356 from zsxwing/SPARK-4481 and squashes the following commits:
76a9891 [zsxwing] Add a note that keys may be added or removed
0ebc42a [zsxwing] Fix the wrong description of updateFunc
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3bf7ceeb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3bf7ceeb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3bf7ceeb
Branch: refs/heads/master
Commit: 3bf7ceebb10741a8b32e0c00f0edfd3a222ec5cd
Parents: 22fc4e7
Author: zsxwing <zs...@gmail.com>
Authored: Wed Nov 19 13:17:15 2014 -0800
Committer: Tathagata Das <ta...@gmail.com>
Committed: Wed Nov 19 13:17:15 2014 -0800
----------------------------------------------------------------------
.../streaming/dstream/PairDStreamFunctions.scala | 16 +++++++---------
1 file changed, 7 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/3bf7ceeb/streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala
----------------------------------------------------------------------
diff --git a/streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala b/streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala
index b39f47f..3f03f42 100644
--- a/streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala
+++ b/streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala
@@ -398,10 +398,9 @@ class PairDStreamFunctions[K, V](self: DStream[(K,V)])
* Return a new "state" DStream where the state for each key is updated by applying
* the given function on the previous state of the key and the new values of each key.
* org.apache.spark.Partitioner is used to control the partitioning of each RDD.
- * @param updateFunc State update function. If `this` function returns None, then
- * corresponding state key-value pair will be eliminated. Note, that
- * this function may generate a different a tuple with a different key
- * than the input key. It is up to the developer to decide whether to
+ * @param updateFunc State update function. Note, that this function may generate a different
+ * tuple with a different key than the input key. Therefore keys may be removed
+ * or added in this way. It is up to the developer to decide whether to
* remember the partitioner despite the key being changed.
* @param partitioner Partitioner for controlling the partitioning of each RDD in the new
* DStream
@@ -442,11 +441,10 @@ class PairDStreamFunctions[K, V](self: DStream[(K,V)])
* Return a new "state" DStream where the state for each key is updated by applying
* the given function on the previous state of the key and the new values of each key.
* org.apache.spark.Partitioner is used to control the partitioning of each RDD.
- * @param updateFunc State update function. If `this` function returns None, then
- * corresponding state key-value pair will be eliminated. Note, that
- * this function may generate a different a tuple with a different key
- * than the input key. It is up to the developer to decide whether to
- * remember the partitioner despite the key being changed.
+ * @param updateFunc State update function. Note, that this function may generate a different
+ * tuple with a different key than the input key. Therefore keys may be removed
+ * or added in this way. It is up to the developer to decide whether to
+ * remember the partitioner despite the key being changed.
* @param partitioner Partitioner for controlling the partitioning of each RDD in the new
* DStream
* @param rememberPartitioner Whether to remember the paritioner object in the generated RDDs.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org