You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by matuskik <gi...@git.apache.org> on 2015/03/03 16:15:33 UTC
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
GitHub user matuskik opened a pull request:
https://github.com/apache/spark/pull/4875
[SPARK-6139] [Streaming] Allow pre-populate sliding window with initial ...
Each new computed window in WindowedDStream checks for empty slots and fills them with the initial list of supplied RDDs.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kikinteractive/spark kik-changes
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4875.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4875
----
commit 7f428350358b573caf1852d4f88d159d9f86cb5d
Author: Matus Faro <ma...@kik.com>
Date: 2015-03-03T15:14:08Z
[SPARK-6139] [Streaming] Allow pre-populate sliding window with initial data
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-77644400
[Test build #28356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28356/consoleFull) for PR 4875 at commit [`3916512`](https://github.com/apache/spark/commit/3916512afc1f4d6994cef5728820002c4cb9126d).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/4875#discussion_r25982687
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/WindowedDStream.scala ---
@@ -62,7 +65,22 @@ class WindowedDStream[T: ClassTag](
override def compute(validTime: Time): Option[RDD[T]] = {
val currentWindow = new Interval(validTime - windowDuration + parent.slideDuration, validTime)
- val rddsInWindow = parent.slice(currentWindow)
+ val parentRddsInWindow = parent.slice(currentWindow)
+ val rddsInWindow = _initialWindow match {
+ case None => {
+ parentRddsInWindow
+ }
+ case Some (initialWindowRDD) => {
+ if(parentRddsInWindow.length - windowBatchCount == 0) {
--- End diff --
These condition can be simplified isnt it?
```
val rddsInWindow = if (parentRddsInWindow.length < windowBatchCount && !initialWindow.exists) { ... } else { ... }
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-77303929
Does Spark Streaming's checkpoint fail to work for recovery?
IIUC from your description you want to recover the window using external data (Cassandra), rather than recover from checkpoint data, and current checkpoint is mutable to the code change, is that right?
Maybe this is a way to solve your problem, but I'm not sure is there any other solution can solve your problem. Feeding into the source InputDStream probably is another way compared to change the `WindowedDStream` code, like feeding again from Kafka from the last offset, so you don't need to save the data into another DB. Currently I've no concrete idea about this,
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-76964290
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-97365488
In that case, mind closing this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/4875#discussion_r25746891
--- Diff: streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java ---
@@ -160,6 +160,38 @@ public void testWindowWithSlideDuration() {
@SuppressWarnings("unchecked")
@Test
+ public void testWindowWithInitialData()
+ {
+
+ List<JavaRDD<Integer>> initialData = Arrays.asList(
+ ssc.sparkContext().parallelize(Arrays.asList(-3, -2, -1)),
+ ssc.sparkContext().parallelize(Arrays.asList(1, 2, 3)),
+ ssc.sparkContext().parallelize(Arrays.asList(4, 5, 6)));
+
+ List<List<Integer>> inputData = Arrays.asList(
+ Arrays.asList(7, 8, 9),
+ Arrays.asList(10, 11, 12),
+ Arrays.asList(13, 14, 15),
+ Arrays.asList(16, 17, 18),
+ Arrays.asList(19, 20, 21),
+ Arrays.asList(22, 23, 24));
+
+ List<List<Integer>> expected = Arrays.asList(
+ Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12),
+ Arrays.asList(7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18),
+ Arrays.asList(13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24),
+ Arrays.asList(19, 20, 21, 22, 23, 24));
+
+ JavaDStream<Integer> stream = JavaTestUtils.attachTestInputStream(ssc, inputData, 1);
+ JavaDStream<Integer> windowed = stream.window(new Duration(4000), new Duration(2000), Optional.of(initialData));
--- End diff --
I think this line is > 100.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by matuskik <gi...@git.apache.org>.
Github user matuskik commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-79385275
@tdas, I haven't forgotten about this PR, I will make the appropriate changes at some time in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-77644121
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-77655289
[Test build #28356 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28356/consoleFull) for PR 4875 at commit [`3916512`](https://github.com/apache/spark/commit/3916512afc1f4d6994cef5728820002c4cb9126d).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by matuskik <gi...@git.apache.org>.
Github user matuskik closed the pull request at:
https://github.com/apache/spark/pull/4875
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/4875#discussion_r25982750
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/WindowedDStream.scala ---
@@ -62,7 +65,22 @@ class WindowedDStream[T: ClassTag](
override def compute(validTime: Time): Option[RDD[T]] = {
val currentWindow = new Interval(validTime - windowDuration + parent.slideDuration, validTime)
- val rddsInWindow = parent.slice(currentWindow)
+ val parentRddsInWindow = parent.slice(currentWindow)
+ val rddsInWindow = _initialWindow match {
+ case None => {
+ parentRddsInWindow
+ }
+ case Some (initialWindowRDD) => {
+ if(parentRddsInWindow.length - windowBatchCount == 0) {
+ // Window is full
+ parentRddsInWindow
+ } else {
+ // Fill the remainder of the window with initial data
+ parentRddsInWindow ++ initialWindowRDD.drop(
--- End diff --
The ordering of the RDDs in the union gets messed up with this. Prepend the initialRdds to the parentRdds
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/4875#discussion_r25981446
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala ---
@@ -649,7 +649,23 @@ abstract class DStream[T: ClassTag] (
* DStream's batching interval
*/
def window(windowDuration: Duration, slideDuration: Duration): DStream[T] = {
- new WindowedDStream(this, windowDuration, slideDuration)
+ new WindowedDStream(this, windowDuration, slideDuration, None)
+ }
+
+ /**
+ * Return a new DStream in which each RDD contains all the elements in seen in a
+ * sliding window of time over this DStream.
+ * @param windowDuration width of the window; must be a multiple of this DStream's
+ * batching interval
+ * @param slideDuration sliding interval of the window (i.e., the interval after which
+ * the new DStream will generate RDDs); must be a multiple of this
+ * DStream's batching interval
+ * @param initialWindow initial window values to prepend starting with the oldest entry
--- End diff --
It would be good to specify in the docs, that initial window is an ordered list of RDDs, and each RDD represents data from a batch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-96769443
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-77083222
Hi @matuskik , what is your basic usage scenario of this window with initial data?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by matuskik <gi...@git.apache.org>.
Github user matuskik commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-77179421
@jerryshao, I am in the process of developing my application but basically I have a stream of events that I persist outside of Spark in Cassandra and also fed into a window. Whenever I re-submit my application and it fails to load from a checkpoint due to changes, I load the data from Cassandra and initialize the window to previous state and continue streaming where I left off. The reason is I have a large window (1-7 days) and I can't wait a day for the window to fill up as my application will be changing fairly often. I would welcome any other suggestions you may have.
Also, if you can think of a more efficient way to implement this, let me know, especially in terms of garbage collecting the initial data after it goes out of scope in the window. Do you think it would make more sense from `WindowedDStream` to feed the initial data into the parent `DStream`'s `generatedRDDs`? I am still wrapping my head around what exactly is executed on the driver and worker and this is my first time writing Scala so any help is appreciated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-77655296
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28356/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/4875#discussion_r25980986
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaDStream.scala ---
@@ -83,6 +86,24 @@ class JavaDStream[T](val dstream: DStream[T])(implicit val classTag: ClassTag[T]
dstream.window(windowDuration, slideDuration)
/**
+ * Return a new DStream in which each RDD contains all the elements in seen in a
+ * sliding window of time over this DStream.
+ * @param windowDuration width of the window; must be a multiple of this DStream's
+ * batching interval
+ * @param slideDuration sliding interval of the window (i.e., the interval after which
+ * the new DStream will generate RDDs); must be a multiple of this
+ * DStream's batching interval
+ * @param initialWindow initial window values to prepend starting with the oldest entry
+ */
+ def window(windowDuration: Duration,
--- End diff --
Same question as below, why optional?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-77644096
This is a good idea. However the PR needs a bit more work.
1. You have add unit tests in the WindowOperationsSuite in Scala. The tests should verify the results for multiple batches as the window slides over all the initial RDDs. Also boundary conditions like what happens if you give 10 initial RDDs but the window = 5x the batch interval (the first 6 initial RDDs should be ignored and never processed). The Java test looks fine.
2. Please add the python API as well. And python unit tests similar to the Scala to ensure python does not mess up the ordering of initial rdds, etc.
3. You probably have to add binary compatibility exclusions. I am starting the test but it will fail. The output of the test will tell you what to add in the MimaExcludes.scala.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by matuskik <gi...@git.apache.org>.
Github user matuskik commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-96771177
Unfortunately I will not have time to complete this pull request as I've moved away from this approach. You can either close this PR or take it over.
Keep in mind, the current code has a problem with conflicting Partitioners. If the initialWindow has a different Partitioner than the Partitioner for the stream, the `PartitionerAwareUnionRDD` will throw an `ArrayIndexOutOfBoundsException`. I don't have the exact stack trace available anymore.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by matuskik <gi...@git.apache.org>.
Github user matuskik commented on the pull request:
https://github.com/apache/spark/pull/4875#issuecomment-77586406
@jerryshao:
- Yes, the purpose is to be able to recover when a checkpoint recovery is not possible.
- I agree, re-streaming a day worth of data into the window would also fill the window. The problem is that the day worth of data would have to be re-streamed on app start up in one batch and it would persist in the window for a whole day before going out of scope. This means the window would at one point hold double the amount of data (two days) before the one big batch of data goes out of scope.
So what I'm doing in my application is creating `CassandraJavaRDD`'s via the `spark-cassandra-connector` library and passing these RDDs as the initial window data.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/4875#discussion_r25980966
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala ---
@@ -101,6 +101,23 @@ class JavaPairDStream[K, V](val dstream: DStream[(K, V)])(
dstream.window(windowDuration, slideDuration)
/**
+ * Return a new DStream which is computed based on windowed batches of this DStream.
+ * @param windowDuration duration (i.e., width) of the window;
+ * must be a multiple of this DStream's interval
+ * @param slideDuration sliding interval of the window (i.e., the interval after which
+ * the new DStream will generate RDDs); must be a multiple of this
+ * DStream's interval
+ * @param initialWindow initial window values to prepend starting with the oldest entry
+ */
+ def window(windowDuration: Duration,
+ slideDuration: Duration,
+ initialWindow: Optional[JList[JavaPairRDD[K, V]]]): JavaPairDStream[K, V] =
--- End diff --
Why is this an option? There is already a version of window where you dont have to specify the initial RDDs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6139] [Streaming] Allow pre-populate sl...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/4875#discussion_r25981054
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala ---
@@ -649,7 +649,23 @@ abstract class DStream[T: ClassTag] (
* DStream's batching interval
*/
def window(windowDuration: Duration, slideDuration: Duration): DStream[T] = {
- new WindowedDStream(this, windowDuration, slideDuration)
+ new WindowedDStream(this, windowDuration, slideDuration, None)
+ }
+
+ /**
+ * Return a new DStream in which each RDD contains all the elements in seen in a
+ * sliding window of time over this DStream.
+ * @param windowDuration width of the window; must be a multiple of this DStream's
+ * batching interval
+ * @param slideDuration sliding interval of the window (i.e., the interval after which
+ * the new DStream will generate RDDs); must be a multiple of this
+ * DStream's batching interval
+ * @param initialWindow initial window values to prepend starting with the oldest entry
+ */
+ def window(windowDuration: Duration,
+ slideDuration: Duration,
+ initialWindow: Option[Seq[RDD[T]]]): DStream[T] = {
--- End diff --
Same question, why option.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org