You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tdas <gi...@git.apache.org> on 2015/10/24 00:30:05 UTC
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/9256
[SPARK-2629][STREAMING] Basic implementation of trackStateByKey
Current updateStateByKey provides stateful processing in Spark Streaming. It allows the user to maintain per-key state and manage that state using an updateFunction. The updateFunction is called for each key, and it uses new data and existing state of the key, to generate an updated state. However, based on community feedback, we have learnt the following lessons.
* Need for more optimized state management that does not scan every key
* Need to make it easier to implement common use cases - (a) timeout of idle data, (b) returning items other than state
The high level idea that of this PR
* Introduce a new API trackStateByKey that, allows the user to update per-key state, and emit arbitrary records. The new API is necessary as this will have significantly different semantics than the existing updateStateByKey API. This API will have direct support for timeouts.
* Internally, the system will keep the state data as a map/list within the partitions of the state RDDs. The new data RDDs will be partitioned appropriately, and for all the key-value data, it will lookup the map/list in the state RDD partition and create a new list/map of updated state data. The new state RDD partition will be created based on the update data and if necessary, with old data.
Here is the detailed design doc. Please take a look and provide feedback as comments.
https://docs.google.com/document/d/1NoALLyd83zGs1hNGMm0Pc5YOVgiPpMHugGMk6COqxxE/edit#heading=h.ph3w0clkd4em
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark trackStateByKey
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9256.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9256
----
commit 7f15e29a4c2500cadbd9fd38a7aca5e58e2b9487
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-10-08T08:22:22Z
First draft of sessionByKey
commit ff7731207dc415f951b9451d9b9d3aa2c7ef99cf
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-10-09T01:46:11Z
Renamed SessionMap to SessionStore and fixed checkpointing bug in SessionRDD
commit 1fea358dda31ebc5bf089e95f12a9c56165e4555
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-10-10T00:25:28Z
Added OpenHashMapBasedSessionStore
commit 27dbabc8e3d44467fdc10f2333ef42d59c6f85d8
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-10-13T05:15:56Z
Fixed bugs
commit 3fc50675b3471c14abded7cbfd23693c81c2b58d
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-10-13T05:42:52Z
Made delta chain threshold configurable
commit 514eb017406d531bba41070323cd51efe213ccad
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-10-13T06:18:05Z
Fixed NPE
commit d5b2bec581cccb97b27878b39914b1e5327c3f86
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-10-13T13:22:38Z
consolidation while checkpointing
commit 58eee1ee9f4960b13df084202ae2ed57e671a744
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-10-13T13:40:26Z
Fixed bug
commit 672e3e620774aec13e8b81549a0d7bdbb98f8455
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-10-13T14:21:27Z
optimized serialization
commit 51465f4bea5da3afe6067f05ce2a1bc965b277b5
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-10-22T01:52:06Z
Updated API based on updated design
commit 10f6a0ecbb56e8aaad50681398bfbda7e1134f92
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-10-23T02:41:42Z
Refactoring the code
commit b7c653d164536fc1954597451915221b39aae9f3
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-10-23T22:23:51Z
Added licenses, and fixed stuff
commit bd9cd9449a15914b410a4482d610ce9355f28b54
Author: Tathagata Das <ta...@gmail.com>
Date: 2015-10-23T22:24:41Z
Merge remote-tracking branch 'apache-github/master' into trackStateByKey
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43095278
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/TrackStateSpec.scala ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{HashPartitioner, Partitioner}
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * Abstract class having all the specifications of DStream.trackStateByKey().
+ * Use the `TrackStateSpec.create()` or `TrackStateSpec.create()` to create instances of this class.
+ *
+ * {{{
+ * TrackStateSpec(trackingFunction) // in Scala
+ * TrackStateSpec.create(trackingFunction) // in Java
+ * }}}
+ */
+sealed abstract class TrackStateSpec[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag]
+ extends Serializable {
+
+ def initialState(rdd: RDD[(K, S)]): this.type
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ def numPartitions(numPartitions: Int): this.type
+ def partitioner(partitioner: Partitioner): this.type
+
+ def timeout(interval: Duration): this.type
+}
+
+
+/** Builder object for creating instances of TrackStateSpec */
+object TrackStateSpec {
+
+ def apply[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackingFunction: (K, Option[V], State[S]) => Option[T]): TrackStateSpec[K, V, S, T] = {
+ new TrackStateSpecImpl[K, V, S, T](trackingFunction)
+ }
+
+ def create[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackingFunction: (K, Option[V], State[S]) => Option[T]): TrackStateSpec[K, V, S, T] = {
+ apply(trackingFunction)
+ }
--- End diff --
Yeah, I was planning to add all the Java friendly stuff on in a later PR, and focus on core functionality in this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43224039
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/TrackeStateDStream.scala ---
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import java.io.{IOException, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.util.StateMap
+import org.apache.spark.util.Utils
+
+private[streaming] case class TrackStateRDDRecord[K: ClassTag, S: ClassTag, T: ClassTag](
+ stateMap: StateMap[K, S], emittedRecords: Seq[T])
+
+
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[dstream] var previousSessionRDDPartition: Partition = null
+ private[dstream] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ _sc: SparkContext,
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (K, Option[V], State[S]) => Option[T],
+ currentTime: Long, timeoutThresholdTime: Option[Long]
+ ) extends RDD[TrackStateRDDRecord[K, S, T]](
+ _sc,
+ List(
+ new OneToOneDependency[TrackStateRDDRecord[K, S, T]](prevStateRDD),
+ new OneToOneDependency(partitionedDataRDD))
+ ) {
+
+ @volatile private var doFullScan = false
+
+ require(partitionedDataRDD.partitioner.nonEmpty)
+ require(partitionedDataRDD.partitioner == prevStateRDD.partitioner)
+
+ override val partitioner = prevStateRDD.partitioner
+
+ override def checkpoint(): Unit = {
+ super.checkpoint()
+ doFullScan = true
+ }
+
+ override def compute(
+ partition: Partition, context: TaskContext): Iterator[TrackStateRDDRecord[K, S, T]] = {
+
+ val stateRDDPartition = partition.asInstanceOf[TrackStateRDDPartition]
+ val prevStateRDDIterator = prevStateRDD.iterator(
+ stateRDDPartition.previousSessionRDDPartition, context)
+ val dataIterator = partitionedDataRDD.iterator(
+ stateRDDPartition.partitionedDataRDDPartition, context)
+ if (!prevStateRDDIterator.hasNext) {
+ throw new SparkException(s"Could not find state map in previous state RDD")
+ }
+
+ val newStateMap = prevStateRDDIterator.next().stateMap.copy()
+ val emittedRecords = new ArrayBuffer[T]
+
+ val wrappedState = new StateImpl[S]()
+
+ dataIterator.foreach { case (key, value) =>
+ wrappedState.wrap(newStateMap.get(key))
+ val emittedRecord = trackingFunction(key, Some(value), wrappedState)
+ if (wrappedState.isRemoved) {
+ newStateMap.remove(key)
+ } else if (wrappedState.isUpdated) {
+ newStateMap.put(key, wrappedState.get(), currentTime)
+ }
+ emittedRecords ++= emittedRecord
--- End diff --
I don't think being an iterator cannot be persisted in memory and get reused many times. Oppositely many RDD is designed as iterator like `HadoopRDD`, whereas can be persisted in memory and get used many times. Actually if RDD is chosen to be persisted in memory, it will materialize the iterator by using `unroll` in `MemoryStore`, materialization will be happened at that step, after that this RDD can be reused without recomputing (you can check the code of `CacheManager`) .
I think the problem here is that it is hard to use the iterator because you have to return a combo of (StateMap, emitted records), where StateMap is a materialized one in memory, so being emitted records as iterator is hard to do. Maybe we could refactor this part of code to make it more iterator friendly, otherwise this unmanaged ArrayBuffer will potentially be a OOM issue. Just my thoughts, any misunderstandings please point out :smile: .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44095780
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/TrackStateSpec.scala ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{HashPartitioner, Partitioner}
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * Abstract class having all the specifications of DStream.trackStateByKey().
+ * Use the `TrackStateSpec.create()` or `TrackStateSpec.create()` to create instances of this class.
+ *
+ * {{{
+ * TrackStateSpec(trackingFunction) // in Scala
+ * TrackStateSpec.create(trackingFunction) // in Java
+ * }}}
+ */
+sealed abstract class TrackStateSpec[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag]
+ extends Serializable {
+
+ def initialState(rdd: RDD[(K, S)]): this.type
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ def numPartitions(numPartitions: Int): this.type
+ def partitioner(partitioner: Partitioner): this.type
+
+ def timeout(interval: Duration): this.type
--- End diff --
Spark Streaming Duration. To keep it consistent with other API
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-150709188
**[Test build #44266 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44266/consoleFull)** for PR 9256 at commit [`bd9cd94`](https://github.com/apache/spark/commit/bd9cd9449a15914b410a4482d610ce9355f28b54).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43088697
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+/**
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] and
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream]].
+ * {{{
+ *
+ * }}}
+ */
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise wise it will throw an exception.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value. Note that you cannot update the state if the state is
+ * timing out (that is, `isTimingOut() return true`, or if the state has already been removed by
+ * `remove()`.
+ */
+ def update(newState: S): Unit
+
+ /** Remove the state if it exists. */
+ def remove(): Unit
+
+ /** Is the state going to be timed out by the system after this batch interval */
+ def isTimingOut(): Boolean
+
+ @inline final def getOption(): Option[S] = Option(get())
+
+ /** Get the state if it exists, otherwise return the default value */
+ @inline final def getOrElse[S1 >: S](default: => S1): S1 = {
+ if (exists) this.get else default
+ }
+
+ @inline final override def toString() = getOption.map { _.toString }.getOrElse("<state not set>")
+}
+
+/** Internal implementation of the [[State]] interface */
+private[streaming] class StateImpl[S] extends State[S] {
+
+ private var state: S = null.asInstanceOf[S]
+ private var defined: Boolean = true
+ private var timingOut: Boolean = false
+ private var updated: Boolean = false
+ private var removed: Boolean = false
+
+ // ========= Public API =========
+ def exists(): Boolean = {
+ defined
+ }
+
+ def get(): S = {
+ state
+ }
+
+ def update(newState: S): Unit = {
+ require(!removed, "Cannot update the state after it has been removed")
+ require(!timingOut, "Cannot update the state that is timing out")
+ state = newState
--- End diff --
Is this required for defensive guard `require(!updated, "cannot update the state this is already updated")`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44245704
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
--- End diff --
I am cool with what matei said - function on the object, not on the class. just want to point out that the original confusion raised by @pwendell -- not having `function` (i.e. setters for function) on StateSpec class was confusing for him. @pwendell do you think, with the updated docs, is it still a concern?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44205403
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+/**
+ * :: Experimental ::
+ * DStream representing the stream of records emitted after the `trackStateByKey` operation
+ * on a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] in Scala.
+ * Additionally, it also gives access to the stream of state snapshots, that is, the state data of
+ * all keys after a batch has updated them.
+ *
+ * @tparam K Class of the state key
+ * @tparam S Class of the state data
+ * @tparam T Class of the emitted records
+ */
+@Experimental
+sealed abstract class EmittedRecordsDStream[K, S, T: ClassTag](
+ ssc: StreamingContext) extends DStream[T](ssc) {
+
+ /** Return a pair DStream where each RDD is the snapshot of the state of all the keys. */
+ def stateSnapshots(): DStream[(K, S)]
+}
+
+/** Internal implementation of the [[EmittedRecordsDStream]] */
+private[streaming] class EmittedRecordsDStreamImpl[
+ K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackStateDStream: TrackStateDStream[K, V, S, T])
+ extends EmittedRecordsDStream[K, S, T](trackStateDStream.context) {
+
+ override def slideDuration: Duration = trackStateDStream.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(trackStateDStream)
+
+ override def compute(validTime: Time): Option[RDD[T]] = {
+ trackStateDStream.getOrCompute(validTime).map { _.flatMap[T] { _.emittedRecords } }
+ }
+
+ def stateSnapshots(): DStream[(K, S)] = {
+ trackStateDStream.flatMap[(K, S)] {
+ _.stateMap.getAll().map { case (k, s, _) => (k, s) }.toTraversable }
+ }
+}
+
+/**
+ * A DStream that allows per-key state to be maintains, and arbitrary records to be generated
+ * based on updates to the state.
+ *
+ * @param parent Parent (key, value) stream that is the source
+ * @param spec Specifications of the trackStateByKey operation
+ * @tparam K Key type
+ * @tparam V Value type
+ * @tparam S Type of the state maintained
+ * @tparam T Type of the eiitted records
+ */
+private[streaming] class TrackStateDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ parent: DStream[(K, V)], spec: StateSpecImpl[K, V, S, T])
+ extends DStream[TrackStateRDDRecord[K, S, T]](parent.context) {
+
+ persist(StorageLevel.MEMORY_ONLY)
+
+ private val partitioner = spec.getPartitioner().getOrElse(
+ new HashPartitioner(ssc.sc.defaultParallelism))
+
+ private val trackingFunction = spec.getFunction()
+
+ override def slideDuration: Duration = parent.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(parent)
+
+ override val mustCheckpoint = true
+
+ /** Method that generates a RDD for the given time */
+ override def compute(validTime: Time): Option[RDD[TrackStateRDDRecord[K, S, T]]] = {
+ val prevStateRDD = getOrCompute(validTime - slideDuration).getOrElse {
+ TrackStateRDD.createFromPairRDD[K, V, S, T](
+ spec.getInitialStateRDD().getOrElse(new EmptyRDD[(K, S)](ssc.sparkContext)),
+ partitioner,
+ validTime.milliseconds
+ )
+ }
+ val newDataRDD = parent.getOrCompute(validTime).get
+ val partitionedDataRDD = newDataRDD.partitionBy(partitioner)
+ val timeoutThresholdTime = spec.getTimeoutInterval().map { interval =>
+ (validTime - interval).milliseconds
--- End diff --
Not really. If the batch interval is 1 minute, one can specify 39 seconds as the threshold. And states that have been idle more >= 39 seconds will be removed at some point. The system (currently) does not guarantee exactly when the idle states will be marked for timeout. Now I am considering whether to document that or not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by huitseeker <gi...@git.apache.org>.
Github user huitseeker commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44188027
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/TrackStateRDD.scala ---
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.rdd
+
+import java.io.{IOException, ObjectInputStream, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark.rdd.{MapPartitionsRDD, RDD}
+import org.apache.spark.streaming.{StateImpl, State}
+import org.apache.spark.streaming.util.{EmptyStateMap, StateMap}
+import org.apache.spark.util.Utils
+import org.apache.spark._
+
+private[streaming] case class TrackStateRDDRecord[K, S, T](
+ var stateMap: StateMap[K, S], var emittedRecords: Seq[T]) {
+ /*
+ private def writeObject(outputStream: ObjectOutputStream): Unit = {
+ outputStream.writeObject(stateMap)
+ outputStream.writeInt(emittedRecords.size)
+ val iterator = emittedRecords.iterator
+ while(iterator.hasNext) {
+ outputStream.writeObject(iterator.next)
+ }
+ }
+
+ private def readObject(inputStream: ObjectInputStream): Unit = {
+ stateMap = inputStream.readObject().asInstanceOf[StateMap[K, S]]
+ val numEmittedRecords = inputStream.readInt()
+ val array = new Array[T](numEmittedRecords)
+ var i = 0
+ while(i < numEmittedRecords) {
+ array(i) = inputStream.readObject().asInstanceOf[T]
+ }
+ emittedRecords = array.toSeq
+ }*/
+}
+
+
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[rdd] var previousSessionRDDPartition: Partition = null
+ private[rdd] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+
+/**
+ * RDD storing the keyed-state of trackStateByKey and corresponding emitted records.
+ * Each partition of this RDD has a single record that contains a StateMap storing
+ */
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (K, Option[V], State[S]) => Option[T],
+ currentTime: Long, timeoutThresholdTime: Option[Long]
+ ) extends RDD[TrackStateRDDRecord[K, S, T]](
+ partitionedDataRDD.sparkContext,
+ List(
+ new OneToOneDependency[TrackStateRDDRecord[K, S, T]](prevStateRDD),
+ new OneToOneDependency(partitionedDataRDD))
+ ) {
+
+ @volatile private var doFullScan = false
+
+ require(prevStateRDD.partitioner.nonEmpty)
+ require(partitionedDataRDD.partitioner == prevStateRDD.partitioner)
+
+ override val partitioner = prevStateRDD.partitioner
+
+ override def checkpoint(): Unit = {
+ super.checkpoint()
+ doFullScan = true
+ }
+
+ override def compute(
+ partition: Partition, context: TaskContext): Iterator[TrackStateRDDRecord[K, S, T]] = {
+
+ val stateRDDPartition = partition.asInstanceOf[TrackStateRDDPartition]
+ val prevStateRDDIterator = prevStateRDD.iterator(
+ stateRDDPartition.previousSessionRDDPartition, context)
+ val dataIterator = partitionedDataRDD.iterator(
+ stateRDDPartition.partitionedDataRDDPartition, context)
+
+ val newStateMap = if (prevStateRDDIterator.hasNext) {
+ prevStateRDDIterator.next().stateMap.copy()
+ } else {
+ new EmptyStateMap[K, S]()
+ }
+
+ val emittedRecords = new ArrayBuffer[T]
+
+ val wrappedState = new StateImpl[S]()
+
+ dataIterator.foreach { case (key, value) =>
+ wrappedState.wrap(newStateMap.get(key))
+ val emittedRecord = trackingFunction(key, Some(value), wrappedState)
+ if (wrappedState.isRemoved) {
+ newStateMap.remove(key)
+ } else if (wrappedState.isUpdated) {
+ newStateMap.put(key, wrappedState.get(), currentTime)
+ }
+ emittedRecords ++= emittedRecord
+ }
+
+ if (doFullScan) {
+ if (timeoutThresholdTime.isDefined) {
--- End diff --
Maybe merge those ifs ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44214012
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data and
+ * @tparam KeyType Class of the keys
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ new StateSpecImpl[KeyType, ValueType, StateType, EmittedType](trackingFunction)
+ }
+
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data and
+ * @tparam KeyType Class of the keys
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def create[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ apply(trackingFunction)
+ }
+}
+
+
+/** Internal implementation of [[org.apache.spark.streaming.StateSpec]] interface. */
+private[streaming]
+case class StateSpecImpl[K, V, S, T](
--- End diff --
Again I'm not sure why this is separate from StateSpec if that's just a "data" class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44246742
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+/**
+ * :: Experimental ::
+ * DStream representing the stream of records emitted after the `trackStateByKey` operation
+ * on a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] in Scala.
+ * Additionally, it also gives access to the stream of state snapshots, that is, the state data of
+ * all keys after a batch has updated them.
+ *
+ * @tparam K Class of the state key
+ * @tparam S Class of the state data
+ * @tparam T Class of the emitted records
+ */
+@Experimental
+sealed abstract class EmittedRecordsDStream[K, S, T: ClassTag](
--- End diff --
Sounds too generic to me, and its hard to associate `EmittedDStream` with `trackStateByKey`. Let me throw in another suggestion - `TrackStateDStream`, dstream generated by `trackStateByKey`. Easy to associate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44504184
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala ---
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.util
+
+import java.io.{ObjectInputStream, ObjectOutputStream}
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.SparkConf
+import org.apache.spark.streaming.util.OpenHashMapBasedStateMap._
+import org.apache.spark.util.collection.OpenHashMap
+
+/** Internal interface for defining the map that keeps track of sessions. */
+private[streaming] abstract class StateMap[K: ClassTag, S: ClassTag] extends Serializable {
+
+ /** Get the state for a key if it exists */
+ def get(key: K): Option[S]
+
+ /** Get all the keys and states whose updated time is older than the given threshold time */
+ def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)]
+
+ /** Get all the keys and states in this map. */
+ def getAll(): Iterator[(K, S, Long)]
+
+ /** Add or update state */
+ def put(key: K, state: S, updatedTime: Long): Unit
+
+ /** Remove a key */
+ def remove(key: K): Unit
+
+ /**
+ * Shallow copy `this` map to create a new state map.
+ * Updates to the new map should not mutate `this` map.
+ */
+ def copy(): StateMap[K, S]
+
+ def toDebugString(): String = toString()
+}
+
+/** Companion object for [[StateMap]], with utility methods */
+private[streaming] object StateMap {
+ def empty[K: ClassTag, S: ClassTag]: StateMap[K, S] = new EmptyStateMap[K, S]
+
+ def create[K: ClassTag, S: ClassTag](conf: SparkConf): StateMap[K, S] = {
+ val deltaChainThreshold = conf.getInt("spark.streaming.sessionByKey.deltaChainThreshold",
+ DELTA_CHAIN_LENGTH_THRESHOLD)
+ new OpenHashMapBasedStateMap[K, S](64, deltaChainThreshold)
+ }
+}
+
+/** Implementation of StateMap interface representing an empty map */
+private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMap[K, S] {
+ override def put(key: K, session: S, updateTime: Long): Unit = {
+ throw new NotImplementedError("put() should not be called on an EmptyStateMap")
+ }
+ override def get(key: K): Option[S] = None
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = Iterator.empty
+ override def getAll(): Iterator[(K, S, Long)] = Iterator.empty
+ override def copy(): StateMap[K, S] = this
+ override def remove(key: K): Unit = { }
+ override def toDebugString(): String = ""
+}
+
+/** Implementation of StateMap based on Spark's [[org.apache.spark.util.collection.OpenHashMap]] */
+private[streaming] class OpenHashMapBasedStateMap[K: ClassTag, S: ClassTag](
+ @transient @volatile var parentStateMap: StateMap[K, S],
+ initialCapacity: Int = 64,
+ deltaChainThreshold: Int = DELTA_CHAIN_LENGTH_THRESHOLD
+ ) extends StateMap[K, S] { self =>
+
+ def this(initialCapacity: Int, deltaChainThreshold: Int) = this(
+ new EmptyStateMap[K, S],
+ initialCapacity = initialCapacity,
+ deltaChainThreshold = deltaChainThreshold)
+
+ def this(deltaChainThreshold: Int) = this(
+ initialCapacity = 64, deltaChainThreshold = deltaChainThreshold)
+
+ def this() = this(DELTA_CHAIN_LENGTH_THRESHOLD)
+
+ @transient @volatile private var deltaMap =
+ new OpenHashMap[K, StateInfo[S]](initialCapacity)
+
+ /** Get the session data if it exists */
+ override def get(key: K): Option[S] = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ if (!stateInfo.deleted) {
+ Some(stateInfo.data)
+ } else {
+ None
+ }
+ } else {
+ parentStateMap.get(key)
+ }
+ }
+
+ /** Get all the keys and states whose updated time is older than the give threshold time */
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = {
+ val oldStates = parentStateMap.getByTime(threshUpdatedTime).filter { case (key, value, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.filter { case (_, stateInfo) =>
+ !stateInfo.deleted && stateInfo.updateTime < threshUpdatedTime
+ }.map { case (key, stateInfo) =>
+ (key, stateInfo.data, stateInfo.updateTime)
+ }
+ oldStates ++ updatedStates
+ }
+
+ /** Get all the keys and states in this map. */
+ override def getAll(): Iterator[(K, S, Long)] = {
+
+ val oldStates = parentStateMap.getAll().filter { case (key, _, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.filter { ! _._2.deleted }.map { case (key, stateInfo) =>
+ (key, stateInfo.data, stateInfo.updateTime)
+ }
+ oldStates ++ updatedStates
+ }
+
+ /** Add or update state */
+ override def put(key: K, state: S, updateTime: Long): Unit = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ stateInfo.update(state, updateTime)
+ } else {
+ deltaMap.update(key, new StateInfo(state, updateTime))
+ }
+ }
+
+ /** Remove a state */
+ override def remove(key: K): Unit = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ stateInfo.markDeleted()
+ } else {
+ val newInfo = new StateInfo[S](deleted = true)
+ deltaMap.update(key, newInfo)
+ }
+ }
+
+ /**
+ * Shallow copy the map to create a new session store. Updates to the new map
+ * should not mutate `this` map.
+ */
+ override def copy(): StateMap[K, S] = {
+ new OpenHashMapBasedStateMap[K, S](this, deltaChainThreshold = deltaChainThreshold)
+ }
+
+ /** Whether the delta chain lenght is long enough that it should be compacted */
+ def shouldCompact: Boolean = {
+ deltaChainLength >= deltaChainThreshold
+ }
+
+ /** Length of the delta chains of this map */
+ def deltaChainLength: Int = parentStateMap match {
+ case map: OpenHashMapBasedStateMap[_, _] => map.deltaChainLength + 1
+ case _ => 0
+ }
+
+ /**
+ * Approximate number of keys in the map. This is an overestimation that is mainly used to
+ * reserve capacity in a new map at delta compaction time.
+ */
+ def approxSize: Int = deltaMap.size + {
+ parentStateMap match {
+ case s: OpenHashMapBasedStateMap[_, _] => s.approxSize
+ case _ => 0
+ }
+ }
+
+ /** Get all the data of this map as string formatted as a tree based on the delta depth */
+ override def toDebugString(): String = {
+ val tabs = if (deltaChainLength > 0) {
+ (" " * (deltaChainLength - 1)) + "+--- "
+ } else ""
+ parentStateMap.toDebugString() + "\n" + deltaMap.iterator.mkString(tabs, "\n" + tabs, "")
+ }
+
+ override def toString(): String = {
+ s"[${System.identityHashCode(this)}, ${System.identityHashCode(parentStateMap)}]"
+ }
+
+ /**
+ * Serialize the map data. Besides serialization, this method actually compact the deltas
+ * (if needed) in a single pass over all the data in the map.
+ */
+
+ private def writeObject(outputStream: ObjectOutputStream): Unit = {
+ // Write all the non-transient fields, especially class tags, etc.
+ outputStream.defaultWriteObject()
+
+ // Write the data in the delta of this state map
+ outputStream.writeInt(deltaMap.size)
+ val deltaMapIterator = deltaMap.iterator
+ var deltaMapCount = 0
+ while (deltaMapIterator.hasNext) {
+ deltaMapCount += 1
+ val (key, stateInfo) = deltaMapIterator.next()
+ outputStream.writeObject(key)
+ outputStream.writeObject(stateInfo)
+ }
+ assert(deltaMapCount == deltaMap.size)
+
+ // Write the data in the parent state map while copying the data into a new parent map for
+ // compaction (if needed)
+ val doCompaction = shouldCompact
+ val newParentSessionStore = if (doCompaction) {
+ val initCapacity = if (approxSize > 0) approxSize else 64
+ new OpenHashMapBasedStateMap[K, S](initialCapacity = initCapacity, deltaChainThreshold)
+ } else { null }
+
+ val iterOfActiveSessions = parentStateMap.getAll()
+
+ var parentSessionCount = 0
+
+ // First write the approximate size of the data to be written, so that readObject can
+ // allocate appropriately sized OpenHashMap.
+ outputStream.writeInt(approxSize)
+
+ while(iterOfActiveSessions.hasNext) {
+ parentSessionCount += 1
+
+ val (key, state, updateTime) = iterOfActiveSessions.next()
+ outputStream.writeObject(key)
+ outputStream.writeObject(state)
+ outputStream.writeLong(updateTime)
+
+ if (doCompaction) {
+ newParentSessionStore.deltaMap.update(
+ key, StateInfo(state, updateTime, deleted = false))
+ }
+ }
+
+ // Write the final limit marking object with the correct count of records written.
+ val limiterObj = new LimitMarker(parentSessionCount)
+ outputStream.writeObject(limiterObj)
+ if (doCompaction) {
+ parentStateMap = newParentSessionStore
+ }
+ }
+
+ /** Deserialize the map data. */
+ private def readObject(inputStream: ObjectInputStream): Unit = {
+
+ // Read the non-transient fields, especially class tags, etc.
+ inputStream.defaultReadObject()
+
+ // Read the data of the delta
+ val deltaMapSize = inputStream.readInt()
+ deltaMap = new OpenHashMap[K, StateInfo[S]]()
--- End diff --
nit: `deltaMapSize` can be sent as `initialCapacity`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153299106
**[Test build #44911 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44911/consoleFull)** for PR 9256 at commit [`df927ba`](https://github.com/apache/spark/commit/df927bada90b58e75616b55564adc37eb83292f4).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `sealed abstract class State[S] `\n * `case class TrackStateSpecImpl[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](`\n * `abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](`\n * ` case class StateInfo[S](`\n * ` class LimitMarker(val num: Int) extends Serializable`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-154859165
**[Test build #45310 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45310/consoleFull)** for PR 9256 at commit [`a78130d`](https://github.com/apache/spark/commit/a78130d502576d437e859261137bad67e15725ae).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `sealed abstract class State[S] `\n * `sealed abstract class StateSpec[K, V, S, T] extends Serializable `\n * `case class StateSpecImpl[K, V, S, T](`\n * `sealed abstract class EmittedRecordsDStream[K, S, T: ClassTag](`\n * ` case class StateInfo[S](`\n * ` class LimitMarker(val num: Int) extends Serializable`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155575106
**[Test build #45550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45550/consoleFull)** for PR 9256 at commit [`77c9a66`](https://github.com/apache/spark/commit/77c9a66e911adf74014bd9b16fb26153a445d372).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155594095
**[Test build #45550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45550/consoleFull)** for PR 9256 at commit [`77c9a66`](https://github.com/apache/spark/commit/77c9a66e911adf74014bd9b16fb26153a445d372).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `sealed abstract class State[S] `\n * `sealed abstract class StateSpec[KeyType, ValueType, StateType, EmittedType] extends Serializable `\n * `case class StateSpecImpl[K, V, S, T](`\n * `sealed abstract class TrackStateDStream[KeyType, StateType, EmittedType: ClassTag](`\n * `class InternalTrackStateDStream[K: ClassTag, V: ClassTag, S: ClassTag, E: ClassTag](`\n * ` case class StateInfo[S](`\n * ` class LimitMarker(val num: Int) extends Serializable`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44197458
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala ---
@@ -351,6 +351,50 @@ class PairDStreamFunctions[K, V](self: DStream[(K, V)])
}
/**
+ * :: Experimental ::
+ * Return a new DStream of data generated by combining the key-value data in `this` stream
+ * with a continuously updated per-key state. The user-provided state tracking function is
+ * applied on each keyed data item along with its corresponding state. The function can choose to
+ * update/remove the state and return a transformed data, which forms the
+ * [[org.apache.spark.streaming.dstream.EmittedRecordsDStream]].
+ *
+ * The specifications of this transformation is made through the
+ * [[org.apache.spark.streaming.StateSpec StateSpec]] class. Besides the tracking function, there
+ * are a number of optional parameters - initial state data, number of partitions, timeouts, etc.
+ * See the [[org.apache.spark.streaming.StateSpec StateSpec spec docs]] for more details.
+ *
+ * Scala example of using `trackStateByKey`:
+ * {{{
+ * def trackingFunction(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ * // Check state exists, accordingly update/remove state and return data to emit
+ * }
+ *
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ *
+ * }}}
+ *
+ * Java example of using `trackStateByKey`:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
+ *
+ * @tparam StateType
+ */
+ @Experimental
+ def trackStateByKey[StateType: ClassTag, EmittedType: ClassTag](
+ spec: StateSpec[K, V, StateType, StateType]): EmittedRecordsDStream[K, StateType, EmittedType] = {
--- End diff --
nit: spec: StateSpec[K, V, StateType, `StateType`] -> spec: StateSpec[K, V, StateType, `EmittedType`]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153001030
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44808/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153000205
**[Test build #44807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44807/consoleFull)** for PR 9256 at commit [`be8cffc`](https://github.com/apache/spark/commit/be8cffc6ccddb14d993564669f380c300e83dc6b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-154859167
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-154429147
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153309419
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155551252
**[Test build #45545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45545/consoleFull)** for PR 9256 at commit [`f1a6696`](https://github.com/apache/spark/commit/f1a669653811d6558b692fc521a0a7f29972ba33).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44095987
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+
+abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ ssc: StreamingContext) extends DStream[T](ssc) {
+
+ def stateSnapshots(): DStream[(K, S)]
+}
+
+
+private[streaming] class EmittedRecordsDStreamImpl[
+ K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackStateDStream: TrackStateDStream[K, V, S, T])
+ extends EmittedRecordsDStream[K, V, S, T](trackStateDStream.context) {
+
+ override def slideDuration: Duration = trackStateDStream.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(trackStateDStream)
+
+ override def compute(validTime: Time): Option[RDD[T]] = {
+ trackStateDStream.getOrCompute(validTime).map { _.flatMap[T] { _.emittedRecords } }
+ }
+
+ def stateSnapshots(): DStream[(K, S)] = {
+ trackStateDStream.flatMap[(K, S)] {
+ _.stateMap.getAll().map { case (k, s, _) => (k, s) }.toTraversable }
+ }
+}
+
+/**
+ * A DStream that allows per-key state to be maintains, and arbitrary records to be generated
+ * based on updates to the state.
+ *
+ * @param parent Parent (key, value) stream that is the source
+ * @param spec Specifications of the trackStateByKey operation
+ * @tparam K Key type
+ * @tparam V Value type
+ * @tparam S Type of the state maintained
+ * @tparam T Type of the eiitted records
+ */
+private[streaming] class TrackStateDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ parent: DStream[(K, V)], spec: TrackStateSpecImpl[K, V, S, T])
+ extends DStream[TrackStateRDDRecord[K, S, T]](parent.context) {
+
+ persist(StorageLevel.MEMORY_ONLY)
+
+ private val partitioner = spec.getPartitioner().getOrElse(
+ new HashPartitioner(ssc.sc.defaultParallelism))
+
+ private val trackingFunction = spec.getFunction()
+
+ override def slideDuration: Duration = parent.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(parent)
+
+ override val mustCheckpoint = true
+
+ /** Method that generates a RDD for the given time */
+ override def compute(validTime: Time): Option[RDD[TrackStateRDDRecord[K, S, T]]] = {
+ val prevStateRDD = getOrCompute(validTime - slideDuration).getOrElse {
+ TrackStateRDD.createFromPairRDD[K, V, S, T](
+ spec.getInitialStateRDD().getOrElse(new EmptyRDD[(K, S)](ssc.sparkContext)),
+ partitioner,
+ validTime.milliseconds
+ )
+ }
+ val newDataRDD = parent.getOrCompute(validTime).get
--- End diff --
In practice it really cannot, but yeah its better to account for None's
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155437951
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43206878
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/TrackeStateDStream.scala ---
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import java.io.{IOException, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.util.StateMap
+import org.apache.spark.util.Utils
+
+private[streaming] case class TrackStateRDDRecord[K: ClassTag, S: ClassTag, T: ClassTag](
+ stateMap: StateMap[K, S], emittedRecords: Seq[T])
+
+
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[dstream] var previousSessionRDDPartition: Partition = null
+ private[dstream] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ _sc: SparkContext,
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (K, Option[V], State[S]) => Option[T],
+ currentTime: Long, timeoutThresholdTime: Option[Long]
+ ) extends RDD[TrackStateRDDRecord[K, S, T]](
+ _sc,
+ List(
+ new OneToOneDependency[TrackStateRDDRecord[K, S, T]](prevStateRDD),
+ new OneToOneDependency(partitionedDataRDD))
+ ) {
+
+ @volatile private var doFullScan = false
+
+ require(partitionedDataRDD.partitioner.nonEmpty)
+ require(partitionedDataRDD.partitioner == prevStateRDD.partitioner)
+
+ override val partitioner = prevStateRDD.partitioner
+
+ override def checkpoint(): Unit = {
+ super.checkpoint()
+ doFullScan = true
+ }
+
+ override def compute(
+ partition: Partition, context: TaskContext): Iterator[TrackStateRDDRecord[K, S, T]] = {
+
+ val stateRDDPartition = partition.asInstanceOf[TrackStateRDDPartition]
+ val prevStateRDDIterator = prevStateRDD.iterator(
+ stateRDDPartition.previousSessionRDDPartition, context)
+ val dataIterator = partitionedDataRDD.iterator(
+ stateRDDPartition.partitionedDataRDDPartition, context)
+ if (!prevStateRDDIterator.hasNext) {
+ throw new SparkException(s"Could not find state map in previous state RDD")
+ }
+
+ val newStateMap = prevStateRDDIterator.next().stateMap.copy()
+ val emittedRecords = new ArrayBuffer[T]
+
+ val wrappedState = new StateImpl[S]()
+
+ dataIterator.foreach { case (key, value) =>
+ wrappedState.wrap(newStateMap.get(key))
+ val emittedRecord = trackingFunction(key, Some(value), wrappedState)
+ if (wrappedState.isRemoved) {
+ newStateMap.remove(key)
+ } else if (wrappedState.isUpdated) {
+ newStateMap.put(key, wrappedState.get(), currentTime)
+ }
+ emittedRecords ++= emittedRecord
--- End diff --
I think you have to materialize the emitted records. Since the combo of [StateMap, emitted records] is a single record for the StateRDD, when the "record" is computed AND persisted, all the necessary information needs to be available in memory, and not require recomputing. Once persisted, the record should be usable any number of times. So if the emitted data *inside* that record is an iterator, it can not be reused any number of times. Which breaks the RDD's specifications.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155613167
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44095840
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/TrackStateSpec.scala ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{HashPartitioner, Partitioner}
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * Abstract class having all the specifications of DStream.trackStateByKey().
+ * Use the `TrackStateSpec.create()` or `TrackStateSpec.create()` to create instances of this class.
+ *
+ * {{{
+ * TrackStateSpec(trackingFunction) // in Scala
+ * TrackStateSpec.create(trackingFunction) // in Java
+ * }}}
+ */
+sealed abstract class TrackStateSpec[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag]
+ extends Serializable {
+
+ def initialState(rdd: RDD[(K, S)]): this.type
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ def numPartitions(numPartitions: Int): this.type
+ def partitioner(partitioner: Partitioner): this.type
+
+ def timeout(interval: Duration): this.type
+}
+
+
+/** Builder object for creating instances of TrackStateSpec */
+object TrackStateSpec {
+
+ def apply[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackingFunction: (K, Option[V], State[S]) => Option[T]): TrackStateSpec[K, V, S, T] = {
+ new TrackStateSpecImpl[K, V, S, T](trackingFunction)
+ }
+
+ def create[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackingFunction: (K, Option[V], State[S]) => Option[T]): TrackStateSpec[K, V, S, T] = {
+ apply(trackingFunction)
+ }
--- End diff --
An alternative suggestion from community. Just make an implicit conversion to Option. That would make Scala users happy.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155553722
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45545/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155572357
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by huitseeker <gi...@git.apache.org>.
Github user huitseeker commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-154550377
Great job so far, I like this a lot !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44470724
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.ClosureCleaner
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * def trackingFunction(data: Option[ValueType], wrappedState: State[StateType]): EmittedType = {
+ * ...
+ * }
+ *
+ * val spec = StateSpec.function(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[KeyType, ValueType, StateType, EmittedDataType] spec =
+ * StateStateSpec.function[KeyType, ValueType, StateType, EmittedDataType](trackingFunction)
+ * .numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[KeyType, ValueType, StateType, EmittedType] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey` */
+ def initialState(rdd: RDD[(KeyType, StateType)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey` */
+ def initialState(javaPairRDD: JavaPairRDD[KeyType, StateType]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used.
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * def trackingFunction(data: Option[ValueType], wrappedState: State[StateType]): EmittedType = {
+ * ...
+ * }
+ *
+ * val spec = StateSpec.function(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[KeyType, ValueType, StateType, EmittedDataType] spec =
+ * StateStateSpec.function[KeyType, ValueType, StateType, EmittedDataType](trackingFunction)
+ * .numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data
+ * @tparam KeyType Class of the keys
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def function[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (Time, KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ ClosureCleaner.clean(trackingFunction, checkSerializable = true)
+ new StateSpecImpl(trackingFunction)
+ }
+
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def function[ValueType, StateType, EmittedType](
+ trackingFunction: (Option[ValueType], State[StateType]) => EmittedType
+ ): StateSpec[Any, ValueType, StateType, EmittedType] = {
--- End diff --
`StateSpec[Any, ValueType, StateType, EmittedType]` will require the user to cast the type if he wants to use the key type. Right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43981909
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+
+abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
--- End diff --
I see - i guess it depends what you define as state. I think of `S` as "stored state" and `T` as "emitted state". Maybe that's off? I think `EmittedDStream` could be okay, it's a bit awkward but I think still better than adding a new term.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44095867
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+
+abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ ssc: StreamingContext) extends DStream[T](ssc) {
+
+ def stateSnapshots(): DStream[(K, S)]
--- End diff --
snapshots of what?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44092275
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+/**
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] and
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream]].
+ * {{{
+ *
+ * }}}
+ */
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise wise it will throw an exception.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value. Note that you cannot update the state if the state is
+ * timing out (that is, `isTimingOut() return true`, or if the state has already been removed by
+ * `remove()`.
+ */
+ def update(newState: S): Unit
+
+ /** Remove the state if it exists. */
+ def remove(): Unit
--- End diff --
The only semantically similar thing is a HashMap, and there both Scala and Java HashMap uses remove()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43088193
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+/**
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] and
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream]].
+ * {{{
+ *
+ * }}}
+ */
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise wise it will throw an exception.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value. Note that you cannot update the state if the state is
+ * timing out (that is, `isTimingOut() return true`, or if the state has already been removed by
+ * `remove()`.
+ */
+ def update(newState: S): Unit
+
+ /** Remove the state if it exists. */
+ def remove(): Unit
+
+ /** Is the state going to be timed out by the system after this batch interval */
+ def isTimingOut(): Boolean
+
+ @inline final def getOption(): Option[S] = Option(get())
+
+ /** Get the state if it exists, otherwise return the default value */
+ @inline final def getOrElse[S1 >: S](default: => S1): S1 = {
--- End diff --
Not sure is this "call-by-name" parameter Java friendly? Assuming this `State` should also be used in Java code :).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44202542
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala ---
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.util
+
+import java.io.{ObjectInputStream, ObjectOutputStream}
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.SparkConf
+import org.apache.spark.streaming.util.OpenHashMapBasedStateMap._
+import org.apache.spark.util.collection.OpenHashMap
+
+/** Internal interface for defining the map that keeps track of sessions. */
+private[streaming] abstract class StateMap[K: ClassTag, S: ClassTag] extends Serializable {
+
+ /** Get the state for a key if it exists */
+ def get(key: K): Option[S]
+
+ /** Get all the keys and states whose updated time is older than the given threshold time */
+ def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)]
+
+ /** Get all the keys and states in this map. */
+ def getAll(): Iterator[(K, S, Long)]
+
+ /** Add or update state */
+ def put(key: K, state: S, updatedTime: Long): Unit
+
+ /** Remove a key */
+ def remove(key: K): Unit
+
+ /**
+ * Shallow copy `this` map to create a new state map.
+ * Updates to the new map should not mutate `this` map.
+ */
+ def copy(): StateMap[K, S]
+
+ def toDebugString(): String = toString()
+}
+
+/** Companion object for [[StateMap]], with utility methods */
+private[streaming] object StateMap {
+ def empty[K: ClassTag, S: ClassTag]: StateMap[K, S] = new EmptyStateMap[K, S]
+
+ def create[K: ClassTag, S: ClassTag](conf: SparkConf): StateMap[K, S] = {
+ val deltaChainThreshold = conf.getInt("spark.streaming.sessionByKey.deltaChainThreshold",
+ DELTA_CHAIN_LENGTH_THRESHOLD)
+ new OpenHashMapBasedStateMap[K, S](64, deltaChainThreshold)
+ }
+}
+
+/** Specific implementation of SessionStore interface representing an empty map */
+private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMap[K, S] {
+ override def put(key: K, session: S, updateTime: Long): Unit = {
+ throw new NotImplementedError("put() should not be called on an EmptyStateMap")
+ }
+ override def get(key: K): Option[S] = None
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = Iterator.empty
+ override def getAll(): Iterator[(K, S, Long)] = Iterator.empty
+ override def copy(): StateMap[K, S] = new EmptyStateMap[K, S]
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155553713
**[Test build #45545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45545/consoleFull)** for PR 9256 at commit [`f1a6696`](https://github.com/apache/spark/commit/f1a669653811d6558b692fc521a0a7f29972ba33).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `sealed abstract class State[S] `\n * `sealed abstract class StateSpec[KeyType, ValueType, StateType, EmittedType] extends Serializable `\n * `case class StateSpecImpl[K, V, S, T](`\n * `sealed abstract class TrackStateDStream[KeyType, StateType, EmittedType: ClassTag](`\n * `class InternalTrackStateDStream[K: ClassTag, V: ClassTag, S: ClassTag, E: ClassTag](`\n * ` case class StateInfo[S](`\n * ` class LimitMarker(val num: Int) extends Serializable`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44200358
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala ---
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.util
+
+import java.io.{ObjectInputStream, ObjectOutputStream}
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.SparkConf
+import org.apache.spark.streaming.util.OpenHashMapBasedStateMap._
+import org.apache.spark.util.collection.OpenHashMap
+
+/** Internal interface for defining the map that keeps track of sessions. */
+private[streaming] abstract class StateMap[K: ClassTag, S: ClassTag] extends Serializable {
+
+ /** Get the state for a key if it exists */
+ def get(key: K): Option[S]
+
+ /** Get all the keys and states whose updated time is older than the given threshold time */
+ def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)]
+
+ /** Get all the keys and states in this map. */
+ def getAll(): Iterator[(K, S, Long)]
+
+ /** Add or update state */
+ def put(key: K, state: S, updatedTime: Long): Unit
+
+ /** Remove a key */
+ def remove(key: K): Unit
+
+ /**
+ * Shallow copy `this` map to create a new state map.
+ * Updates to the new map should not mutate `this` map.
+ */
+ def copy(): StateMap[K, S]
+
+ def toDebugString(): String = toString()
+}
+
+/** Companion object for [[StateMap]], with utility methods */
+private[streaming] object StateMap {
+ def empty[K: ClassTag, S: ClassTag]: StateMap[K, S] = new EmptyStateMap[K, S]
+
+ def create[K: ClassTag, S: ClassTag](conf: SparkConf): StateMap[K, S] = {
+ val deltaChainThreshold = conf.getInt("spark.streaming.sessionByKey.deltaChainThreshold",
+ DELTA_CHAIN_LENGTH_THRESHOLD)
+ new OpenHashMapBasedStateMap[K, S](64, deltaChainThreshold)
+ }
+}
+
+/** Specific implementation of SessionStore interface representing an empty map */
+private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMap[K, S] {
+ override def put(key: K, session: S, updateTime: Long): Unit = {
+ throw new NotImplementedError("put() should not be called on an EmptyStateMap")
+ }
+ override def get(key: K): Option[S] = None
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = Iterator.empty
+ override def getAll(): Iterator[(K, S, Long)] = Iterator.empty
+ override def copy(): StateMap[K, S] = new EmptyStateMap[K, S]
+ override def remove(key: K): Unit = { }
+ override def toDebugString(): String = ""
+}
+
+
+
+/** Implementation of StateMap based on Spark's OpenHashMap */
+private[streaming] class OpenHashMapBasedStateMap[K: ClassTag, S: ClassTag](
+ @transient @volatile var parentStateMap: StateMap[K, S],
+ initialCapacity: Int = 64,
+ deltaChainThreshold: Int = DELTA_CHAIN_LENGTH_THRESHOLD
+ ) extends StateMap[K, S] { self =>
+
+ def this(initialCapacity: Int, deltaChainThreshold: Int) = this(
+ new EmptyStateMap[K, S],
+ initialCapacity = initialCapacity,
+ deltaChainThreshold = deltaChainThreshold)
+
+ def this(deltaChainThreshold: Int) = this(
+ initialCapacity = 64, deltaChainThreshold = deltaChainThreshold)
+
+ def this() = this(DELTA_CHAIN_LENGTH_THRESHOLD)
+
+ @transient @volatile private var deltaMap =
+ new OpenHashMap[K, StateInfo[S]](initialCapacity)
+
+ /** Get the session data if it exists */
+ override def get(key: K): Option[S] = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ if (!stateInfo.deleted) {
+ Some(stateInfo.data)
+ } else {
+ None
+ }
+ } else {
+ parentStateMap.get(key)
+ }
+ }
+
+ /** Get all the keys and states whose updated time is older than the give threshold time */
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = {
+ val oldStates = parentStateMap.getByTime(threshUpdatedTime).filter { case (key, value, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.flatMap { case (key, stateInfo) =>
+ if (! stateInfo.deleted && stateInfo.updateTime < threshUpdatedTime) {
+ Some((key, stateInfo.data, stateInfo.updateTime))
+ } else None
+ }
+ oldStates ++ updatedStates
+ }
+
+ /** Get all the keys and states in this map. */
+ override def getAll(): Iterator[(K, S, Long)] = {
+
+ val oldStates = parentStateMap.getAll().filter { case (key, _, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.filter { ! _._2.deleted }.map { case (key, stateInfo) =>
+ (key, stateInfo.data, stateInfo.updateTime)
+ }
+ oldStates ++ updatedStates
+ }
+
+ /** Add or update state */
+ override def put(key: K, state: S, updateTime: Long): Unit = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ stateInfo.update(state, updateTime)
+ } else {
+ deltaMap.update(key, new StateInfo(state, updateTime))
+ }
+ }
+
+ /** Remove a state */
+ override def remove(key: K): Unit = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ stateInfo.markDeleted()
+ } else {
+ val newInfo = new StateInfo[S](deleted = true)
+ deltaMap.update(key, newInfo)
+ }
+ }
+
+ /**
+ * Shallow copy the map to create a new session store. Updates to the new map
+ * should not mutate `this` map.
+ */
+ override def copy(): StateMap[K, S] = {
+ new OpenHashMapBasedStateMap[K, S](this, deltaChainThreshold = deltaChainThreshold)
+ }
+
+ def shouldCompact: Boolean = {
+ deltaChainLength >= deltaChainThreshold
--- End diff --
I did benchmark the cost of deltas, albeit not in rigorous detail. The the get() latency doubles when the delta length is approx 10-20. But I am less worried about that because in the bigger picture RDD checkpointing is the major bottleneck, not these. So we have to solve that first, before solving for optimizing in-memory data structures.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44233204
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala ---
@@ -351,6 +351,50 @@ class PairDStreamFunctions[K, V](self: DStream[(K, V)])
}
/**
+ * :: Experimental ::
+ * Return a new DStream of data generated by combining the key-value data in `this` stream
+ * with a continuously updated per-key state. The user-provided state tracking function is
+ * applied on each keyed data item along with its corresponding state. The function can choose to
+ * update/remove the state and return a transformed data, which forms the
+ * [[org.apache.spark.streaming.dstream.EmittedRecordsDStream]].
+ *
+ * The specifications of this transformation is made through the
+ * [[org.apache.spark.streaming.StateSpec StateSpec]] class. Besides the tracking function, there
+ * are a number of optional parameters - initial state data, number of partitions, timeouts, etc.
+ * See the [[org.apache.spark.streaming.StateSpec StateSpec spec docs]] for more details.
+ *
+ * Scala example of using `trackStateByKey`:
+ * {{{
+ * def trackingFunction(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ * // Check state exists, accordingly update/remove state and return data to emit
+ * }
+ *
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ *
+ * }}}
+ *
+ * Java example of using `trackStateByKey`:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
--- End diff --
Aah, yes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43982371
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala ---
@@ -350,6 +349,18 @@ class PairDStreamFunctions[K, V](self: DStream[(K, V)])
)
}
+ /** TODO: Add scala docs */
+ def trackStateByKey[S: ClassTag, T: ClassTag](
+ spec: TrackStateSpec[K, V, S, T]): EmittedRecordsDStream[K, V, S, T] = {
+ new EmittedRecordsDStreamImpl[K, V, S, T](
+ new TrackStateDStream[K, V, S, T](
+ self,
+ spec.asInstanceOf[TrackStateSpecImpl[K, V, S, T]]
--- End diff --
It doesn't seem like a big deal if the getters are public.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-150709503
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153297944
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44910/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44337618
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.language.implicitConversions
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Scala example of using `State`:
+ * {{{
+ * def trackStateFunc(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ *
+ * // Check if state exists
+ * if (state.exists) {
+ *
+ * val existingState = wrappedState.get // Get the existing state
+ *
+ * val shouldRemove = ... // Decide whether to remove the state
+ *
+ * if (shouldRemove) {
+ *
+ * wrappedState.remove() // Remove the state
+ *
+ * } else {
+ *
+ * val newState = ...
+ * wrappedState(newState) // Set the new state
+ *
+ * }
+ * } else {
+ *
+ * val initialState = ...
+ * state.update(initialState) // Set the initial state
+ *
+ * }
+ * }
+ *
+ * }}}
+ *
+ * Java example:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
+ */
+@Experimental
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise it will throw `java.util.NoSuchElementException`.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ *
+ * @throws java.util.NoSuchElementException If the state does not exist.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ *
+ * @throws java.lang.IllegalArgumentException If the state has already been removed, or is
+ * going to be removed
+ */
+ def update(newState: S): Unit
+
+ /**
+ * Remove the state if it exists.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ */
+ def remove(): Unit
+
+ /**
+ * Whether the state is timing out and going to be removed by the system after the current batch.
+ * This timeou can occur if timeout duration has been specified in the
+ * [[org.apache.spark.streaming.StateSpec StatSpec]] and the key has not received any new data
+ * for that timeout duration.
+ */
+ def isTimingOut(): Boolean
+
+ /**
+ * Get the state as an [[scala.Option]]. It will be `Some(state)` if it exists, otherwise `None`.
+ */
+ @inline final def getOption(): Option[S] = if (exists) Some(get()) else None
+
+ @inline final override def toString(): String = {
+ getOption.map { _.toString }.getOrElse("<state not set>")
+ }
+}
+
+private[streaming]
+object State {
+ implicit def toOption[S](state: State[S]): Option[S] = state.getOption()
+}
--- End diff --
Cool. Will remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by huitseeker <gi...@git.apache.org>.
Github user huitseeker commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44187957
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/TrackStateRDD.scala ---
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.rdd
+
+import java.io.{IOException, ObjectInputStream, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark.rdd.{MapPartitionsRDD, RDD}
+import org.apache.spark.streaming.{StateImpl, State}
+import org.apache.spark.streaming.util.{EmptyStateMap, StateMap}
+import org.apache.spark.util.Utils
+import org.apache.spark._
+
+private[streaming] case class TrackStateRDDRecord[K, S, T](
+ var stateMap: StateMap[K, S], var emittedRecords: Seq[T]) {
+ /*
+ private def writeObject(outputStream: ObjectOutputStream): Unit = {
+ outputStream.writeObject(stateMap)
+ outputStream.writeInt(emittedRecords.size)
+ val iterator = emittedRecords.iterator
+ while(iterator.hasNext) {
+ outputStream.writeObject(iterator.next)
+ }
+ }
+
+ private def readObject(inputStream: ObjectInputStream): Unit = {
+ stateMap = inputStream.readObject().asInstanceOf[StateMap[K, S]]
+ val numEmittedRecords = inputStream.readInt()
+ val array = new Array[T](numEmittedRecords)
+ var i = 0
+ while(i < numEmittedRecords) {
+ array(i) = inputStream.readObject().asInstanceOf[T]
+ }
+ emittedRecords = array.toSeq
+ }*/
+}
+
+
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[rdd] var previousSessionRDDPartition: Partition = null
+ private[rdd] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+
+/**
+ * RDD storing the keyed-state of trackStateByKey and corresponding emitted records.
+ * Each partition of this RDD has a single record that contains a StateMap storing
+ */
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (K, Option[V], State[S]) => Option[T],
+ currentTime: Long, timeoutThresholdTime: Option[Long]
+ ) extends RDD[TrackStateRDDRecord[K, S, T]](
+ partitionedDataRDD.sparkContext,
+ List(
+ new OneToOneDependency[TrackStateRDDRecord[K, S, T]](prevStateRDD),
+ new OneToOneDependency(partitionedDataRDD))
+ ) {
+
+ @volatile private var doFullScan = false
+
+ require(prevStateRDD.partitioner.nonEmpty)
+ require(partitionedDataRDD.partitioner == prevStateRDD.partitioner)
+
+ override val partitioner = prevStateRDD.partitioner
+
+ override def checkpoint(): Unit = {
+ super.checkpoint()
+ doFullScan = true
+ }
+
+ override def compute(
+ partition: Partition, context: TaskContext): Iterator[TrackStateRDDRecord[K, S, T]] = {
+
+ val stateRDDPartition = partition.asInstanceOf[TrackStateRDDPartition]
+ val prevStateRDDIterator = prevStateRDD.iterator(
+ stateRDDPartition.previousSessionRDDPartition, context)
+ val dataIterator = partitionedDataRDD.iterator(
+ stateRDDPartition.partitionedDataRDDPartition, context)
+
+ val newStateMap = if (prevStateRDDIterator.hasNext) {
+ prevStateRDDIterator.next().stateMap.copy()
+ } else {
+ new EmptyStateMap[K, S]()
+ }
+
+ val emittedRecords = new ArrayBuffer[T]
+
+ val wrappedState = new StateImpl[S]()
+
+ dataIterator.foreach { case (key, value) =>
+ wrappedState.wrap(newStateMap.get(key))
+ val emittedRecord = trackingFunction(key, Some(value), wrappedState)
+ if (wrappedState.isRemoved) {
+ newStateMap.remove(key)
+ } else if (wrappedState.isUpdated) {
+ newStateMap.put(key, wrappedState.get(), currentTime)
+ }
+ emittedRecords ++= emittedRecord
+ }
+
+ if (doFullScan) {
+ if (timeoutThresholdTime.isDefined) {
+ newStateMap.getByTime(timeoutThresholdTime.get).foreach { case (key, state, _) =>
+ wrappedState.wrapTiminoutState(state)
+ val emittedRecord = trackingFunction(key, None, wrappedState)
+ emittedRecords ++= emittedRecord
+ newStateMap.remove(key)
+ }
+ }
+ }
+
+ Iterator(TrackStateRDDRecord(newStateMap, emittedRecords))
+ }
+
+ override protected def getPartitions: Array[Partition] = {
+ Array.tabulate(prevStateRDD.partitions.length) { i =>
+ new TrackStateRDDPartition(i, prevStateRDD, partitionedDataRDD)}
+ }
+
+ override def clearDependencies() {
--- End diff --
I'd rather see [procedure syntax avoided](http://docs.scala-lang.org/style/declarations.html).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44303122
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data and
+ * @tparam KeyType Class of the keys
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
--- End diff --
We could have multiple versions of the function() method that take different function types. That seems fine to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44205625
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala ---
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.util
+
+import java.io.{ObjectInputStream, ObjectOutputStream}
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.SparkConf
+import org.apache.spark.streaming.util.OpenHashMapBasedStateMap._
+import org.apache.spark.util.collection.OpenHashMap
+
+/** Internal interface for defining the map that keeps track of sessions. */
+private[streaming] abstract class StateMap[K: ClassTag, S: ClassTag] extends Serializable {
+
+ /** Get the state for a key if it exists */
+ def get(key: K): Option[S]
+
+ /** Get all the keys and states whose updated time is older than the given threshold time */
+ def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)]
+
+ /** Get all the keys and states in this map. */
+ def getAll(): Iterator[(K, S, Long)]
+
+ /** Add or update state */
+ def put(key: K, state: S, updatedTime: Long): Unit
+
+ /** Remove a key */
+ def remove(key: K): Unit
+
+ /**
+ * Shallow copy `this` map to create a new state map.
+ * Updates to the new map should not mutate `this` map.
+ */
+ def copy(): StateMap[K, S]
+
+ def toDebugString(): String = toString()
+}
+
+/** Companion object for [[StateMap]], with utility methods */
+private[streaming] object StateMap {
+ def empty[K: ClassTag, S: ClassTag]: StateMap[K, S] = new EmptyStateMap[K, S]
+
+ def create[K: ClassTag, S: ClassTag](conf: SparkConf): StateMap[K, S] = {
+ val deltaChainThreshold = conf.getInt("spark.streaming.sessionByKey.deltaChainThreshold",
+ DELTA_CHAIN_LENGTH_THRESHOLD)
+ new OpenHashMapBasedStateMap[K, S](64, deltaChainThreshold)
+ }
+}
+
+/** Specific implementation of SessionStore interface representing an empty map */
+private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMap[K, S] {
+ override def put(key: K, session: S, updateTime: Long): Unit = {
+ throw new NotImplementedError("put() should not be called on an EmptyStateMap")
+ }
+ override def get(key: K): Option[S] = None
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = Iterator.empty
+ override def getAll(): Iterator[(K, S, Long)] = Iterator.empty
+ override def copy(): StateMap[K, S] = new EmptyStateMap[K, S]
+ override def remove(key: K): Unit = { }
+ override def toDebugString(): String = ""
+}
+
+
+
+/** Implementation of StateMap based on Spark's OpenHashMap */
+private[streaming] class OpenHashMapBasedStateMap[K: ClassTag, S: ClassTag](
+ @transient @volatile var parentStateMap: StateMap[K, S],
+ initialCapacity: Int = 64,
+ deltaChainThreshold: Int = DELTA_CHAIN_LENGTH_THRESHOLD
+ ) extends StateMap[K, S] { self =>
+
+ def this(initialCapacity: Int, deltaChainThreshold: Int) = this(
+ new EmptyStateMap[K, S],
+ initialCapacity = initialCapacity,
+ deltaChainThreshold = deltaChainThreshold)
+
+ def this(deltaChainThreshold: Int) = this(
+ initialCapacity = 64, deltaChainThreshold = deltaChainThreshold)
+
+ def this() = this(DELTA_CHAIN_LENGTH_THRESHOLD)
+
+ @transient @volatile private var deltaMap =
+ new OpenHashMap[K, StateInfo[S]](initialCapacity)
+
+ /** Get the session data if it exists */
+ override def get(key: K): Option[S] = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ if (!stateInfo.deleted) {
+ Some(stateInfo.data)
+ } else {
+ None
+ }
+ } else {
+ parentStateMap.get(key)
+ }
+ }
+
+ /** Get all the keys and states whose updated time is older than the give threshold time */
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = {
+ val oldStates = parentStateMap.getByTime(threshUpdatedTime).filter { case (key, value, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.flatMap { case (key, stateInfo) =>
+ if (! stateInfo.deleted && stateInfo.updateTime < threshUpdatedTime) {
+ Some((key, stateInfo.data, stateInfo.updateTime))
+ } else None
+ }
+ oldStates ++ updatedStates
+ }
+
+ /** Get all the keys and states in this map. */
+ override def getAll(): Iterator[(K, S, Long)] = {
+
+ val oldStates = parentStateMap.getAll().filter { case (key, _, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.filter { ! _._2.deleted }.map { case (key, stateInfo) =>
+ (key, stateInfo.data, stateInfo.updateTime)
+ }
+ oldStates ++ updatedStates
+ }
+
+ /** Add or update state */
+ override def put(key: K, state: S, updateTime: Long): Unit = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ stateInfo.update(state, updateTime)
+ } else {
+ deltaMap.update(key, new StateInfo(state, updateTime))
+ }
+ }
+
+ /** Remove a state */
+ override def remove(key: K): Unit = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ stateInfo.markDeleted()
+ } else {
+ val newInfo = new StateInfo[S](deleted = true)
+ deltaMap.update(key, newInfo)
+ }
+ }
+
+ /**
+ * Shallow copy the map to create a new session store. Updates to the new map
+ * should not mutate `this` map.
+ */
+ override def copy(): StateMap[K, S] = {
+ new OpenHashMapBasedStateMap[K, S](this, deltaChainThreshold = deltaChainThreshold)
+ }
+
+ def shouldCompact: Boolean = {
+ deltaChainLength >= deltaChainThreshold
+ }
+
+ def deltaChainLength: Int = parentStateMap match {
+ case map: OpenHashMapBasedStateMap[_, _] => map.deltaChainLength + 1
+ case _ => 0
+ }
+
+ def approxSize: Int = deltaMap.size + {
+ parentStateMap match {
+ case s: OpenHashMapBasedStateMap[_, _] => s.approxSize
+ case _ => 0
+ }
+ }
+
+ override def toDebugString(): String = {
+ val tabs = if (deltaChainLength > 0) {
+ (" " * (deltaChainLength - 1)) + "+--- "
+ } else ""
+ parentStateMap.toDebugString() + "\n" + deltaMap.iterator.mkString(tabs, "\n" + tabs, "")
+ }
+
+ override def toString(): String = {
+ s"[${System.identityHashCode(this)}, ${System.identityHashCode(parentStateMap)}]"
+ }
+
+ private def writeObject(outputStream: ObjectOutputStream): Unit = {
+
+ outputStream.defaultWriteObject()
+
+ // Write the deltaMap
+ outputStream.writeInt(deltaMap.size)
+ val deltaMapIterator = deltaMap.iterator
+ var deltaMapCount = 0
+ while (deltaMapIterator.hasNext) {
+ deltaMapCount += 1
+ val (key, stateInfo) = deltaMapIterator.next()
+ outputStream.writeObject(key)
+ outputStream.writeObject(stateInfo)
+ }
+ assert(deltaMapCount == deltaMap.size)
+
+ // Write the parentStateMap while consolidating
+ val doCompaction = shouldCompact
+ val newParentSessionStore = if (doCompaction) {
+ val initCapacity = if (approxSize > 0) approxSize else 64
+ new OpenHashMapBasedStateMap[K, S](initialCapacity = initCapacity, deltaChainThreshold)
+ } else { null }
+
+ val iterOfActiveSessions = parentStateMap.getAll()
+
+ var parentSessionCount = 0
+
+ outputStream.writeInt(approxSize)
+
+ while(iterOfActiveSessions.hasNext) {
+ parentSessionCount += 1
+
+ val (key, state, updateTime) = iterOfActiveSessions.next()
+ outputStream.writeObject(key)
+ outputStream.writeObject(state)
+ outputStream.writeLong(updateTime)
+
+ if (doCompaction) {
+ newParentSessionStore.deltaMap.update(
+ key, StateInfo(state, updateTime, deleted = false))
+ }
+ }
+ val limiterObj = new LimitMarker(parentSessionCount)
+ outputStream.writeObject(limiterObj)
+ if (doCompaction) {
+ parentStateMap = newParentSessionStore
+ }
+ }
+
+ private def readObject(inputStream: ObjectInputStream): Unit = {
+ inputStream.defaultReadObject()
+
+ val deltaMapSize = inputStream.readInt()
+ deltaMap = new OpenHashMap[K, StateInfo[S]]()
+ var deltaMapCount = 0
+ while (deltaMapCount < deltaMapSize) {
+ val key = inputStream.readObject().asInstanceOf[K]
+ val sessionInfo = inputStream.readObject().asInstanceOf[StateInfo[S]]
+ deltaMap.update(key, sessionInfo)
+ deltaMapCount += 1
+ }
+
+ val parentSessionStoreSizeHint = inputStream.readInt()
+ val newParentSessionStore = new OpenHashMapBasedStateMap[K, S](
+ initialCapacity = parentSessionStoreSizeHint, deltaChainThreshold)
+
+ var parentSessionLoopDone = false
+ while(!parentSessionLoopDone) {
+ val obj = inputStream.readObject()
+ if (obj.isInstanceOf[LimitMarker]) {
+ parentSessionLoopDone = true
+ val expectedCount = obj.asInstanceOf[LimitMarker].num
+ assert(expectedCount == newParentSessionStore.deltaMap.size)
+ } else {
+ val key = obj.asInstanceOf[K]
+ val state = inputStream.readObject().asInstanceOf[S]
+ val updateTime = inputStream.readLong()
+ newParentSessionStore.deltaMap.update(
+ key, StateInfo(state, updateTime, deleted = false))
+ }
+ }
+ parentStateMap = newParentSessionStore
+ }
+}
+
+private[streaming] object OpenHashMapBasedStateMap {
+
+ case class StateInfo[S](
--- End diff --
note to self: add docs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44338358
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data and
+ * @tparam KeyType Class of the keys
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
--- End diff --
Okay, here is my proposal, let me know if anyone has any objections. There be two function signatures to start with.
- Simple: `(value: Option[V], state: State[S]) => T`
- Advanced: `(batchTime: Time, key: K, value: Option[V], state: State[S]) => Option[T]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44091385
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+
+abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ ssc: StreamingContext) extends DStream[T](ssc) {
+
+ def stateSnapshots(): DStream[(K, S)]
+}
+
+
+private[streaming] class EmittedRecordsDStreamImpl[
+ K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackStateDStream: TrackStateDStream[K, V, S, T])
+ extends EmittedRecordsDStream[K, V, S, T](trackStateDStream.context) {
+
+ override def slideDuration: Duration = trackStateDStream.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(trackStateDStream)
+
+ override def compute(validTime: Time): Option[RDD[T]] = {
+ trackStateDStream.getOrCompute(validTime).map { _.flatMap[T] { _.emittedRecords } }
+ }
+
+ def stateSnapshots(): DStream[(K, S)] = {
+ trackStateDStream.flatMap[(K, S)] {
+ _.stateMap.getAll().map { case (k, s, _) => (k, s) }.toTraversable }
+ }
+}
+
+/**
+ * A DStream that allows per-key state to be maintains, and arbitrary records to be generated
+ * based on updates to the state.
+ *
+ * @param parent Parent (key, value) stream that is the source
+ * @param spec Specifications of the trackStateByKey operation
+ * @tparam K Key type
+ * @tparam V Value type
+ * @tparam S Type of the state maintained
+ * @tparam T Type of the eiitted records
+ */
+private[streaming] class TrackStateDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ parent: DStream[(K, V)], spec: TrackStateSpecImpl[K, V, S, T])
+ extends DStream[TrackStateRDDRecord[K, S, T]](parent.context) {
+
+ persist(StorageLevel.MEMORY_ONLY)
+
+ private val partitioner = spec.getPartitioner().getOrElse(
+ new HashPartitioner(ssc.sc.defaultParallelism))
+
+ private val trackingFunction = spec.getFunction()
+
+ override def slideDuration: Duration = parent.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(parent)
+
+ override val mustCheckpoint = true
+
+ /** Method that generates a RDD for the given time */
+ override def compute(validTime: Time): Option[RDD[TrackStateRDDRecord[K, S, T]]] = {
+ val prevStateRDD = getOrCompute(validTime - slideDuration).getOrElse {
+ TrackStateRDD.createFromPairRDD[K, V, S, T](
+ spec.getInitialStateRDD().getOrElse(new EmptyRDD[(K, S)](ssc.sparkContext)),
+ partitioner,
+ validTime.milliseconds
+ )
+ }
+ val newDataRDD = parent.getOrCompute(validTime).get
--- End diff --
`parent.getOrCompute(validTime)` could return `None`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-150708248
PR is out - @pwendell @rxin @mateiz
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44205513
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala ---
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.util
+
+import java.io.{ObjectInputStream, ObjectOutputStream}
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.SparkConf
+import org.apache.spark.streaming.util.OpenHashMapBasedStateMap._
+import org.apache.spark.util.collection.OpenHashMap
+
+/** Internal interface for defining the map that keeps track of sessions. */
+private[streaming] abstract class StateMap[K: ClassTag, S: ClassTag] extends Serializable {
+
+ /** Get the state for a key if it exists */
+ def get(key: K): Option[S]
+
+ /** Get all the keys and states whose updated time is older than the given threshold time */
+ def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)]
+
+ /** Get all the keys and states in this map. */
+ def getAll(): Iterator[(K, S, Long)]
+
+ /** Add or update state */
+ def put(key: K, state: S, updatedTime: Long): Unit
+
+ /** Remove a key */
+ def remove(key: K): Unit
+
+ /**
+ * Shallow copy `this` map to create a new state map.
+ * Updates to the new map should not mutate `this` map.
+ */
+ def copy(): StateMap[K, S]
+
+ def toDebugString(): String = toString()
+}
+
+/** Companion object for [[StateMap]], with utility methods */
+private[streaming] object StateMap {
+ def empty[K: ClassTag, S: ClassTag]: StateMap[K, S] = new EmptyStateMap[K, S]
+
+ def create[K: ClassTag, S: ClassTag](conf: SparkConf): StateMap[K, S] = {
+ val deltaChainThreshold = conf.getInt("spark.streaming.sessionByKey.deltaChainThreshold",
+ DELTA_CHAIN_LENGTH_THRESHOLD)
+ new OpenHashMapBasedStateMap[K, S](64, deltaChainThreshold)
+ }
+}
+
+/** Specific implementation of SessionStore interface representing an empty map */
+private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMap[K, S] {
+ override def put(key: K, session: S, updateTime: Long): Unit = {
+ throw new NotImplementedError("put() should not be called on an EmptyStateMap")
+ }
+ override def get(key: K): Option[S] = None
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = Iterator.empty
+ override def getAll(): Iterator[(K, S, Long)] = Iterator.empty
+ override def copy(): StateMap[K, S] = new EmptyStateMap[K, S]
+ override def remove(key: K): Unit = { }
+ override def toDebugString(): String = ""
+}
+
+
+
+/** Implementation of StateMap based on Spark's OpenHashMap */
+private[streaming] class OpenHashMapBasedStateMap[K: ClassTag, S: ClassTag](
+ @transient @volatile var parentStateMap: StateMap[K, S],
+ initialCapacity: Int = 64,
+ deltaChainThreshold: Int = DELTA_CHAIN_LENGTH_THRESHOLD
+ ) extends StateMap[K, S] { self =>
+
+ def this(initialCapacity: Int, deltaChainThreshold: Int) = this(
+ new EmptyStateMap[K, S],
+ initialCapacity = initialCapacity,
+ deltaChainThreshold = deltaChainThreshold)
+
+ def this(deltaChainThreshold: Int) = this(
+ initialCapacity = 64, deltaChainThreshold = deltaChainThreshold)
+
+ def this() = this(DELTA_CHAIN_LENGTH_THRESHOLD)
+
+ @transient @volatile private var deltaMap =
+ new OpenHashMap[K, StateInfo[S]](initialCapacity)
+
+ /** Get the session data if it exists */
+ override def get(key: K): Option[S] = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ if (!stateInfo.deleted) {
+ Some(stateInfo.data)
+ } else {
+ None
+ }
+ } else {
+ parentStateMap.get(key)
+ }
+ }
+
+ /** Get all the keys and states whose updated time is older than the give threshold time */
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = {
+ val oldStates = parentStateMap.getByTime(threshUpdatedTime).filter { case (key, value, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.flatMap { case (key, stateInfo) =>
--- End diff --
both will exhaust the iterator. I think the concern is just readability. I am changing it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44484988
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.ClosureCleaner
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * def trackingFunction(data: Option[ValueType], wrappedState: State[StateType]): EmittedType = {
+ * ...
+ * }
+ *
+ * val spec = StateSpec.function(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[KeyType, ValueType, StateType, EmittedDataType] spec =
+ * StateStateSpec.function[KeyType, ValueType, StateType, EmittedDataType](trackingFunction)
+ * .numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[KeyType, ValueType, StateType, EmittedType] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey` */
+ def initialState(rdd: RDD[(KeyType, StateType)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey` */
+ def initialState(javaPairRDD: JavaPairRDD[KeyType, StateType]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used.
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * def trackingFunction(data: Option[ValueType], wrappedState: State[StateType]): EmittedType = {
+ * ...
+ * }
+ *
+ * val spec = StateSpec.function(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[KeyType, ValueType, StateType, EmittedDataType] spec =
+ * StateStateSpec.function[KeyType, ValueType, StateType, EmittedDataType](trackingFunction)
+ * .numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data
+ * @tparam KeyType Class of the keys
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def function[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (Time, KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ ClosureCleaner.clean(trackingFunction, checkSerializable = true)
+ new StateSpecImpl(trackingFunction)
+ }
+
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def function[ValueType, StateType, EmittedType](
+ trackingFunction: (Option[ValueType], State[StateType]) => EmittedType
+ ): StateSpec[Any, ValueType, StateType, EmittedType] = {
--- End diff --
But without fixing this, it become more complicated for Java users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44206807
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ *
+ * @param trackingFunction
+ * @tparam KeyType Class of keys
+ * @tparam ValueType
+ * @tparam StateType
+ * @tparam EmittedType
+ * @return
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
--- End diff --
@rxin @pwendell There are three questions that I would like to discuss regarding this.
1. Value for a key being presented in function as Seq or Option. Seq allows multiple values for a key to be presented together so that the user can sort them in the correct order before updating state. But it means we have to do the grouping. Potentially additional cost. My thought is that there are user-level ways to get around it (like groupByKey) and we can add this later if we need.
2. Batch time in the function signature. This is based on some good feedback on the doc. The function would like to know what is the current batch time in which it is being processed and accordingly do its own expiry (if the simple timeout logic is not sufficient). That would make the function signature more complex `(key: Key, value: Option[Value], state: State[S], batchTime: Time) => Option[T]` Currently there is no good workaround other than the user using `DStream.transform` to embed the batch time in every record. Very inefficient.
3. Even if we dont include time the signature `(K, Option[V], State[S]) => Option[T]` is a daunting signature. It might be worthwhilte to include multiple methods - a simplified one like `(Option[V], State[S]) => T` and a full one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153314478
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153309078
**[Test build #44913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44913/consoleFull)** for PR 9256 at commit [`6a75966`](https://github.com/apache/spark/commit/6a75966cf863566008c6c03ad7fe7924e3a25db9).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44095929
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+
+abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
--- End diff --
This is the stream of records T generated after apply the function `(K, Option[V], State[S]) => Option[T]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44233566
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data and
+ * @tparam KeyType Class of the keys
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ new StateSpecImpl[KeyType, ValueType, StateType, EmittedType](trackingFunction)
+ }
+
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data and
+ * @tparam KeyType Class of the keys
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def create[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ apply(trackingFunction)
+ }
+}
+
+
+/** Internal implementation of [[org.apache.spark.streaming.StateSpec]] interface. */
+private[streaming]
+case class StateSpecImpl[K, V, S, T](
--- End diff --
Same reason as `State` - separating public API from private[streaming] ones to make them clearly unaccessible in Java.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44205241
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+/**
+ * :: Experimental ::
+ * DStream representing the stream of records emitted after the `trackStateByKey` operation
+ * on a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] in Scala.
+ * Additionally, it also gives access to the stream of state snapshots, that is, the state data of
+ * all keys after a batch has updated them.
+ *
+ * @tparam K Class of the state key
+ * @tparam S Class of the state data
+ * @tparam T Class of the emitted records
+ */
+@Experimental
+sealed abstract class EmittedRecordsDStream[K, S, T: ClassTag](
--- End diff --
@rxin @pwendell Is `EmittedRecordsDStream` is good or should we change this? Things that have been suggested.
1. EmittedDStream - Pro: shorter, no ref to "record". Con: Emitted what? states?
2. StateDStream - Logic: DStream generated by `trackStateByKey`. Pro: shorter. Con: sounds like a stream of states (similar to updateStateByKey), but that this is not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43947154
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+/**
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] and
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream]].
+ * {{{
+ *
--- End diff --
Todo: Add example.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44299243
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.language.implicitConversions
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Scala example of using `State`:
+ * {{{
+ * def trackStateFunc(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ *
+ * // Check if state exists
+ * if (state.exists) {
+ *
+ * val existingState = wrappedState.get // Get the existing state
+ *
+ * val shouldRemove = ... // Decide whether to remove the state
+ *
+ * if (shouldRemove) {
+ *
+ * wrappedState.remove() // Remove the state
+ *
+ * } else {
+ *
+ * val newState = ...
+ * wrappedState(newState) // Set the new state
+ *
+ * }
+ * } else {
+ *
+ * val initialState = ...
+ * state.update(initialState) // Set the initial state
+ *
+ * }
+ * }
+ *
+ * }}}
+ *
+ * Java example:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
+ */
+@Experimental
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise it will throw `java.util.NoSuchElementException`.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ *
+ * @throws java.util.NoSuchElementException If the state does not exist.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ *
+ * @throws java.lang.IllegalArgumentException If the state has already been removed, or is
+ * going to be removed
+ */
+ def update(newState: S): Unit
+
+ /**
+ * Remove the state if it exists.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ */
+ def remove(): Unit
+
+ /**
+ * Whether the state is timing out and going to be removed by the system after the current batch.
+ * This timeou can occur if timeout duration has been specified in the
+ * [[org.apache.spark.streaming.StateSpec StatSpec]] and the key has not received any new data
+ * for that timeout duration.
+ */
+ def isTimingOut(): Boolean
+
+ /**
+ * Get the state as an [[scala.Option]]. It will be `Some(state)` if it exists, otherwise `None`.
+ */
+ @inline final def getOption(): Option[S] = if (exists) Some(get()) else None
+
+ @inline final override def toString(): String = {
+ getOption.map { _.toString }.getOrElse("<state not set>")
+ }
+}
+
+private[streaming]
+object State {
+ implicit def toOption[S](state: State[S]): Option[S] = state.getOption()
+}
--- End diff --
We shouldn't have that as part of our API, we don't use implicit conversions in other places in Spark. It will be pretty confusing. Instead they can just do state.getOption.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-154858807
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43980418
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala ---
@@ -350,6 +349,18 @@ class PairDStreamFunctions[K, V](self: DStream[(K, V)])
)
}
+ /** TODO: Add scala docs */
+ def trackStateByKey[S: ClassTag, T: ClassTag](
--- End diff --
Can you add the docs? Would make it easier to review the public API's here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44299415
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+/**
+ * :: Experimental ::
+ * DStream representing the stream of records emitted after the `trackStateByKey` operation
+ * on a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] in Scala.
+ * Additionally, it also gives access to the stream of state snapshots, that is, the state data of
+ * all keys after a batch has updated them.
+ *
+ * @tparam K Class of the state key
+ * @tparam S Class of the state data
+ * @tparam T Class of the emitted records
+ */
+@Experimental
+sealed abstract class EmittedRecordsDStream[K, S, T: ClassTag](
--- End diff --
TrackStateDStream sounds good
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44213921
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.language.implicitConversions
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Scala example of using `State`:
+ * {{{
+ * def trackStateFunc(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ *
+ * // Check if state exists
+ * if (state.exists) {
+ *
+ * val existingState = wrappedState.get // Get the existing state
+ *
+ * val shouldRemove = ... // Decide whether to remove the state
+ *
+ * if (shouldRemove) {
+ *
+ * wrappedState.remove() // Remove the state
+ *
+ * } else {
+ *
+ * val newState = ...
+ * wrappedState(newState) // Set the new state
+ *
+ * }
+ * } else {
+ *
+ * val initialState = ...
+ * state.update(initialState) // Set the initial state
+ *
+ * }
+ * }
+ *
+ * }}}
+ *
+ * Java example:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
+ */
+@Experimental
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise it will throw `java.util.NoSuchElementException`.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ *
+ * @throws java.util.NoSuchElementException If the state does not exist.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ *
+ * @throws java.lang.IllegalArgumentException If the state has already been removed, or is
+ * going to be removed
+ */
+ def update(newState: S): Unit
+
+ /**
+ * Remove the state if it exists.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ */
+ def remove(): Unit
+
+ /**
+ * Whether the state is timing out and going to be removed by the system after the current batch.
+ * This timeou can occur if timeout duration has been specified in the
+ * [[org.apache.spark.streaming.StateSpec StatSpec]] and the key has not received any new data
+ * for that timeout duration.
+ */
+ def isTimingOut(): Boolean
+
+ /**
+ * Get the state as an [[scala.Option]]. It will be `Some(state)` if it exists, otherwise `None`.
+ */
+ @inline final def getOption(): Option[S] = if (exists) Some(get()) else None
+
+ @inline final override def toString(): String = {
+ getOption.map { _.toString }.getOrElse("<state not set>")
+ }
+}
+
+private[streaming]
+object State {
+ implicit def toOption[S](state: State[S]): Option[S] = state.getOption()
+}
+
+/** Internal implementation of the [[State]] interface */
+private[streaming] class StateImpl[S] extends State[S] {
--- End diff --
Why did we separate this into another class? I don't see any other subclasses of State or great reasons to create them
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155594325
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153001029
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153000291
**[Test build #44807 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44807/consoleFull)** for PR 9256 at commit [`be8cffc`](https://github.com/apache/spark/commit/be8cffc6ccddb14d993564669f380c300e83dc6b).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `sealed abstract class State[S] `\n * `case class TrackStateSpecImpl[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](`\n * `abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](`\n * ` case class StateInfo[S](`\n * ` class LimitMarker(val num: Int) extends Serializable`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-154859018
**[Test build #45310 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45310/consoleFull)** for PR 9256 at commit [`a78130d`](https://github.com/apache/spark/commit/a78130d502576d437e859261137bad67e15725ae).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155440812
**[Test build #45527 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45527/consoleFull)** for PR 9256 at commit [`fb5a296`](https://github.com/apache/spark/commit/fb5a296ac2c82eec16f2449267f94e98a46c54be).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `sealed abstract class State[S] `\n * `sealed abstract class StateSpec[KeyType, ValueType, StateType, EmittedType] extends Serializable `\n * `case class StateSpecImpl[K, V, S, T](`\n * `sealed abstract class TrackStateDStream[KeyType, StateType, EmittedType: ClassTag](`\n * `class InternalTrackStateDStream[K: ClassTag, V: ClassTag, S: ClassTag, E: ClassTag](`\n * ` case class StateInfo[S](`\n * ` class LimitMarker(val num: Int) extends Serializable`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44091460
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+/**
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] and
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream]].
+ * {{{
+ *
+ * }}}
+ */
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise wise it will throw an exception.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value. Note that you cannot update the state if the state is
+ * timing out (that is, `isTimingOut() return true`, or if the state has already been removed by
+ * `remove()`.
+ */
+ def update(newState: S): Unit
+
+ /** Remove the state if it exists. */
+ def remove(): Unit
+
+ /** Is the state going to be timed out by the system after this batch interval */
+ def isTimingOut(): Boolean
+
+ @inline final def getOption(): Option[S] = Option(get())
+
+ /** Get the state if it exists, otherwise return the default value */
+ @inline final def getOrElse[S1 >: S](default: => S1): S1 = {
+ if (exists) this.get else default
+ }
+
+ @inline final override def toString(): String = {
+ getOption.map { _.toString }.getOrElse("<state not set>")
+ }
+}
+
+/** Internal implementation of the [[State]] interface */
+private[streaming] class StateImpl[S] extends State[S] {
+
+ private var state: S = null.asInstanceOf[S]
+ private var defined: Boolean = true
+ private var timingOut: Boolean = false
+ private var updated: Boolean = false
+ private var removed: Boolean = false
+
+ // ========= Public API =========
+ def exists(): Boolean = {
+ defined
+ }
+
+ def get(): S = {
+ state
--- End diff --
`State.get` says `Get the state if it exists, otherwise wise it will throw an exception.` but here it won't throw an exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43237480
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/TrackeStateDStream.scala ---
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import java.io.{IOException, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.util.StateMap
+import org.apache.spark.util.Utils
+
+private[streaming] case class TrackStateRDDRecord[K: ClassTag, S: ClassTag, T: ClassTag](
+ stateMap: StateMap[K, S], emittedRecords: Seq[T])
+
+
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[dstream] var previousSessionRDDPartition: Partition = null
+ private[dstream] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ _sc: SparkContext,
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (K, Option[V], State[S]) => Option[T],
+ currentTime: Long, timeoutThresholdTime: Option[Long]
+ ) extends RDD[TrackStateRDDRecord[K, S, T]](
+ _sc,
+ List(
+ new OneToOneDependency[TrackStateRDDRecord[K, S, T]](prevStateRDD),
+ new OneToOneDependency(partitionedDataRDD))
+ ) {
+
+ @volatile private var doFullScan = false
+
+ require(partitionedDataRDD.partitioner.nonEmpty)
+ require(partitionedDataRDD.partitioner == prevStateRDD.partitioner)
+
+ override val partitioner = prevStateRDD.partitioner
+
+ override def checkpoint(): Unit = {
+ super.checkpoint()
+ doFullScan = true
+ }
+
+ override def compute(
+ partition: Partition, context: TaskContext): Iterator[TrackStateRDDRecord[K, S, T]] = {
+
+ val stateRDDPartition = partition.asInstanceOf[TrackStateRDDPartition]
+ val prevStateRDDIterator = prevStateRDD.iterator(
+ stateRDDPartition.previousSessionRDDPartition, context)
+ val dataIterator = partitionedDataRDD.iterator(
+ stateRDDPartition.partitionedDataRDDPartition, context)
+ if (!prevStateRDDIterator.hasNext) {
+ throw new SparkException(s"Could not find state map in previous state RDD")
+ }
+
+ val newStateMap = prevStateRDDIterator.next().stateMap.copy()
+ val emittedRecords = new ArrayBuffer[T]
+
+ val wrappedState = new StateImpl[S]()
+
+ dataIterator.foreach { case (key, value) =>
+ wrappedState.wrap(newStateMap.get(key))
+ val emittedRecord = trackingFunction(key, Some(value), wrappedState)
--- End diff --
Yeah. But what if the user actually generates a tuple from upstream code which like ("some-key", null) because he want to signify null downstream. That null could be different from not having any data at all. How does one differentiate that?
I guess it boils down to policy, do we respect nulls in the data or not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155681290
@tdas, look good. My comments are minor. There are some unnecessary Java APIs in this PR. But I can fix them while I'm adding the Java APIs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43095076
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+/**
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] and
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream]].
+ * {{{
+ *
+ * }}}
+ */
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise wise it will throw an exception.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value. Note that you cannot update the state if the state is
+ * timing out (that is, `isTimingOut() return true`, or if the state has already been removed by
+ * `remove()`.
+ */
+ def update(newState: S): Unit
+
+ /** Remove the state if it exists. */
+ def remove(): Unit
+
+ /** Is the state going to be timed out by the system after this batch interval */
+ def isTimingOut(): Boolean
+
+ @inline final def getOption(): Option[S] = Option(get())
+
+ /** Get the state if it exists, otherwise return the default value */
+ @inline final def getOrElse[S1 >: S](default: => S1): S1 = {
--- End diff --
Yeah, that is probably a valid concern. If users cannot call it from Java, this could be a Scala-only thing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by huitseeker <gi...@git.apache.org>.
Github user huitseeker commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44189416
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+/**
+ * :: Experimental ::
+ * DStream representing the stream of records emitted after the `trackStateByKey` operation
+ * on a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] in Scala.
+ * Additionally, it also gives access to the stream of state snapshots, that is, the state data of
+ * all keys after a batch has updated them.
+ *
+ * @tparam K Class of the state key
+ * @tparam S Class of the state data
+ * @tparam T Class of the emitted records
+ */
+@Experimental
+sealed abstract class EmittedRecordsDStream[K, S, T: ClassTag](
+ ssc: StreamingContext) extends DStream[T](ssc) {
+
+ /** Return a pair DStream where each RDD is the snapshot of the state of all the keys. */
+ def stateSnapshots(): DStream[(K, S)]
+}
+
+/** Internal implementation of the [[EmittedRecordsDStream]] */
+private[streaming] class EmittedRecordsDStreamImpl[
+ K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackStateDStream: TrackStateDStream[K, V, S, T])
+ extends EmittedRecordsDStream[K, S, T](trackStateDStream.context) {
+
+ override def slideDuration: Duration = trackStateDStream.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(trackStateDStream)
+
+ override def compute(validTime: Time): Option[RDD[T]] = {
+ trackStateDStream.getOrCompute(validTime).map { _.flatMap[T] { _.emittedRecords } }
+ }
+
+ def stateSnapshots(): DStream[(K, S)] = {
+ trackStateDStream.flatMap[(K, S)] {
+ _.stateMap.getAll().map { case (k, s, _) => (k, s) }.toTraversable }
+ }
+}
+
+/**
+ * A DStream that allows per-key state to be maintains, and arbitrary records to be generated
+ * based on updates to the state.
+ *
+ * @param parent Parent (key, value) stream that is the source
+ * @param spec Specifications of the trackStateByKey operation
+ * @tparam K Key type
+ * @tparam V Value type
+ * @tparam S Type of the state maintained
+ * @tparam T Type of the eiitted records
+ */
+private[streaming] class TrackStateDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ parent: DStream[(K, V)], spec: StateSpecImpl[K, V, S, T])
+ extends DStream[TrackStateRDDRecord[K, S, T]](parent.context) {
+
+ persist(StorageLevel.MEMORY_ONLY)
+
+ private val partitioner = spec.getPartitioner().getOrElse(
+ new HashPartitioner(ssc.sc.defaultParallelism))
+
+ private val trackingFunction = spec.getFunction()
+
+ override def slideDuration: Duration = parent.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(parent)
+
+ override val mustCheckpoint = true
+
+ /** Method that generates a RDD for the given time */
+ override def compute(validTime: Time): Option[RDD[TrackStateRDDRecord[K, S, T]]] = {
+ val prevStateRDD = getOrCompute(validTime - slideDuration).getOrElse {
+ TrackStateRDD.createFromPairRDD[K, V, S, T](
+ spec.getInitialStateRDD().getOrElse(new EmptyRDD[(K, S)](ssc.sparkContext)),
+ partitioner,
+ validTime.milliseconds
+ )
+ }
+ val newDataRDD = parent.getOrCompute(validTime).get
+ val partitionedDataRDD = newDataRDD.partitionBy(partitioner)
+ val timeoutThresholdTime = spec.getTimeoutInterval().map { interval =>
+ (validTime - interval).milliseconds
--- End diff --
The way this variable is used invites intervals that are multiple of the batch interval, and this may be something to document.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155440822
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45527/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-152998884
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44503176
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/TrackStateRDD.scala ---
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.rdd
+
+import java.io.{IOException, ObjectInputStream, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark.rdd.{MapPartitionsRDD, RDD}
+import org.apache.spark.streaming.{Time, StateImpl, State}
+import org.apache.spark.streaming.util.{EmptyStateMap, StateMap}
+import org.apache.spark.util.Utils
+import org.apache.spark._
+
+/**
+ * Record storing the keyed-state [[TrackStateRDD]]. Each record contains a [[StateMap]] and a
+ * sequence of records returned by the tracking function of `trackStateByKey`.
+ */
+private[streaming] case class TrackStateRDDRecord[K, S, T](
+ var stateMap: StateMap[K, S], var emittedRecords: Seq[T])
+
+/**
+ * Partition of the [[TrackStateRDD]], which depends on corresponding partitions of prev state
+ * RDD, and a partitioned keyed-data RDD
+ */
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[rdd] var previousSessionRDDPartition: Partition = null
+ private[rdd] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+
+/**
+ * RDD storing the keyed-state of `trackStateByKey` and corresponding emitted records.
+ * Each partition of this RDD has a single record of type [[TrackStateRDDRecord]]. This contains a
+ * [[StateMap]] (containing the keyed-states) and the sequence of records returned by the tracking
+ * function of `trackStateByKey`.
+ * @param prevStateRDD The previous TrackStateRDD on whose StateMap data `this` RDD will be created
+ * @param partitionedDataRDD The partitioned data RDD which is used update the previous StateMaps
+ * in the `prevStateRDD` to create `this` RDD
+ * @param trackingFunction The function that will be used to update state and return new data
+ * @param batchTime The time of the batch to which this RDD belongs to. Use to update
+ */
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (Time, K, Option[V], State[S]) => Option[T],
+ batchTime: Time, timeoutThresholdTime: Option[Long]
--- End diff --
nit: `timeoutThresholdTime` should be in a new line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153298549
**[Test build #44911 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44911/consoleFull)** for PR 9256 at commit [`df927ba`](https://github.com/apache/spark/commit/df927bada90b58e75616b55564adc37eb83292f4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44094830
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala ---
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.util
+
+import java.io.{ObjectInputStream, ObjectOutputStream}
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.SparkConf
+import org.apache.spark.streaming.util.OpenHashMapBasedStateMap._
+import org.apache.spark.util.collection.OpenHashMap
+
+/** Internal interface for defining the map that keeps track of sessions. */
+private[streaming] abstract class StateMap[K: ClassTag, S: ClassTag] extends Serializable {
+
+ /** Get the state for a key if it exists */
+ def get(key: K): Option[S]
+
+ /** Get all the keys and states whose updated time is older than the given threshold time */
+ def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)]
+
+ /** Get all the keys and states in this map. */
+ def getAll(): Iterator[(K, S, Long)]
+
+ /** Add or update state */
+ def put(key: K, state: S, updatedTime: Long): Unit
+
+ /** Remove a key */
+ def remove(key: K): Unit
+
+ /**
+ * Shallow copy `this` map to create a new state map.
+ * Updates to the new map should not mutate `this` map.
+ */
+ def copy(): StateMap[K, S]
+
+ def toDebugString(): String = toString()
+}
+
+/** Companion object for [[StateMap]], with utility methods */
+private[streaming] object StateMap {
+ def empty[K: ClassTag, S: ClassTag]: StateMap[K, S] = new EmptyStateMap[K, S]
+
+ def create[K: ClassTag, S: ClassTag](conf: SparkConf): StateMap[K, S] = {
+ val deltaChainThreshold = conf.getInt("spark.streaming.sessionByKey.deltaChainThreshold",
+ DELTA_CHAIN_LENGTH_THRESHOLD)
+ new OpenHashMapBasedStateMap[K, S](64, deltaChainThreshold)
+ }
+}
+
+/** Specific implementation of SessionStore interface representing an empty map */
+private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMap[K, S] {
+ override def put(key: K, session: S, updateTime: Long): Unit = {
+ throw new NotImplementedError("put() should not be called on an EmptyStateMap")
+ }
+ override def get(key: K): Option[S] = None
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = Iterator.empty
+ override def getAll(): Iterator[(K, S, Long)] = Iterator.empty
+ override def copy(): StateMap[K, S] = new EmptyStateMap[K, S]
--- End diff --
nit: this method can return `this` since it's immutable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43980731
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+
+abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
--- End diff --
could this be called `EmittedStateDStream`? I don't think the term "Record" has clear semantics here. It might be good to tie it back to the terms already defined (i.e. State).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by huitseeker <gi...@git.apache.org>.
Github user huitseeker commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44187623
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ *
+ * @param trackingFunction
+ * @tparam KeyType Class of keys
+ * @tparam ValueType
+ * @tparam StateType
+ * @tparam EmittedType
+ * @return
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ new StateSpecImpl[KeyType, ValueType, StateType, EmittedType](trackingFunction)
+ }
+
+ def create[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
--- End diff --
Missing `ClassTag`s here as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44198001
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ *
+ * @param trackingFunction
+ * @tparam KeyType Class of keys
+ * @tparam ValueType
+ * @tparam StateType
+ * @tparam EmittedType
+ * @return
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ new StateSpecImpl[KeyType, ValueType, StateType, EmittedType](trackingFunction)
+ }
+
+ def create[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ apply(trackingFunction)
+ }
+}
+
+
+/** Internal implementation of [[org.apache.spark.streaming.StateSpec]] interface. */
+private[streaming]
+case class StateSpecImpl[K, V, S: ClassTag, T: ClassTag](
--- End diff --
You forgot to remove S and T's `ClassTags`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-154858814
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44096879
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+
+abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ ssc: StreamingContext) extends DStream[T](ssc) {
+
+ def stateSnapshots(): DStream[(K, S)]
+}
+
+
+private[streaming] class EmittedRecordsDStreamImpl[
+ K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackStateDStream: TrackStateDStream[K, V, S, T])
+ extends EmittedRecordsDStream[K, V, S, T](trackStateDStream.context) {
+
+ override def slideDuration: Duration = trackStateDStream.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(trackStateDStream)
+
+ override def compute(validTime: Time): Option[RDD[T]] = {
+ trackStateDStream.getOrCompute(validTime).map { _.flatMap[T] { _.emittedRecords } }
+ }
+
+ def stateSnapshots(): DStream[(K, S)] = {
+ trackStateDStream.flatMap[(K, S)] {
+ _.stateMap.getAll().map { case (k, s, _) => (k, s) }.toTraversable }
+ }
+}
+
+/**
+ * A DStream that allows per-key state to be maintains, and arbitrary records to be generated
+ * based on updates to the state.
+ *
+ * @param parent Parent (key, value) stream that is the source
+ * @param spec Specifications of the trackStateByKey operation
+ * @tparam K Key type
+ * @tparam V Value type
+ * @tparam S Type of the state maintained
+ * @tparam T Type of the eiitted records
+ */
+private[streaming] class TrackStateDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ parent: DStream[(K, V)], spec: TrackStateSpecImpl[K, V, S, T])
+ extends DStream[TrackStateRDDRecord[K, S, T]](parent.context) {
+
+ persist(StorageLevel.MEMORY_ONLY)
+
+ private val partitioner = spec.getPartitioner().getOrElse(
+ new HashPartitioner(ssc.sc.defaultParallelism))
+
+ private val trackingFunction = spec.getFunction()
+
+ override def slideDuration: Duration = parent.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(parent)
+
+ override val mustCheckpoint = true
+
+ /** Method that generates a RDD for the given time */
+ override def compute(validTime: Time): Option[RDD[TrackStateRDDRecord[K, S, T]]] = {
+ val prevStateRDD = getOrCompute(validTime - slideDuration).getOrElse {
+ TrackStateRDD.createFromPairRDD[K, V, S, T](
+ spec.getInitialStateRDD().getOrElse(new EmptyRDD[(K, S)](ssc.sparkContext)),
+ partitioner,
+ validTime.milliseconds
+ )
+ }
+ val newDataRDD = parent.getOrCompute(validTime).get
--- End diff --
> In practice it really cannot, but yeah its better to account for None's
A user custom DStream may return `None`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43980438
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/TrackStateSpec.scala ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{HashPartitioner, Partitioner}
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * Abstract class having all the specifications of DStream.trackStateByKey().
+ * Use the `TrackStateSpec.create()` or `TrackStateSpec.create()` to create instances of this class.
+ *
+ * {{{
+ * TrackStateSpec(trackingFunction) // in Scala
+ * TrackStateSpec.create(trackingFunction) // in Java
+ * }}}
+ */
+sealed abstract class TrackStateSpec[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag]
--- End diff --
I would prefer to just call this `StateSpec`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44213932
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.language.implicitConversions
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Scala example of using `State`:
+ * {{{
+ * def trackStateFunc(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ *
+ * // Check if state exists
+ * if (state.exists) {
+ *
+ * val existingState = wrappedState.get // Get the existing state
+ *
+ * val shouldRemove = ... // Decide whether to remove the state
+ *
+ * if (shouldRemove) {
+ *
+ * wrappedState.remove() // Remove the state
+ *
+ * } else {
+ *
+ * val newState = ...
+ * wrappedState(newState) // Set the new state
+ *
+ * }
+ * } else {
+ *
+ * val initialState = ...
+ * state.update(initialState) // Set the initial state
+ *
+ * }
+ * }
+ *
+ * }}}
+ *
+ * Java example:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
+ */
+@Experimental
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise it will throw `java.util.NoSuchElementException`.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ *
+ * @throws java.util.NoSuchElementException If the state does not exist.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ *
+ * @throws java.lang.IllegalArgumentException If the state has already been removed, or is
+ * going to be removed
+ */
+ def update(newState: S): Unit
+
+ /**
+ * Remove the state if it exists.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ */
+ def remove(): Unit
+
+ /**
+ * Whether the state is timing out and going to be removed by the system after the current batch.
+ * This timeou can occur if timeout duration has been specified in the
+ * [[org.apache.spark.streaming.StateSpec StatSpec]] and the key has not received any new data
+ * for that timeout duration.
+ */
+ def isTimingOut(): Boolean
+
+ /**
+ * Get the state as an [[scala.Option]]. It will be `Some(state)` if it exists, otherwise `None`.
+ */
+ @inline final def getOption(): Option[S] = if (exists) Some(get()) else None
+
+ @inline final override def toString(): String = {
+ getOption.map { _.toString }.getOrElse("<state not set>")
+ }
+}
+
+private[streaming]
+object State {
+ implicit def toOption[S](state: State[S]): Option[S] = state.getOption()
+}
+
+/** Internal implementation of the [[State]] interface */
+private[streaming] class StateImpl[S] extends State[S] {
+
+ private var state: S = null.asInstanceOf[S]
+ private var defined: Boolean = true
+ private var timingOut: Boolean = false
+ private var updated: Boolean = false
+ private var removed: Boolean = false
+
+ // ========= Public API =========
+ override def exists(): Boolean = {
+ defined
+ }
+
+ override def get(): S = {
+ state
+ }
+
+ override def update(newState: S): Unit = {
+ require(!removed, "Cannot update the state after it has been removed")
+ require(!timingOut, "Cannot update the state that is timing out")
+ state = newState
+ updated = true
+ }
+
+ override def isTimingOut(): Boolean = {
+ timingOut
+ }
+
+ override def remove(): Unit = {
+ require(!timingOut, "Cannot remove the state that is timing out")
+ require(!removed, "Cannot remove the state that has already been removed")
+ removed = true
+ }
+
+ // ========= Internal API =========
+
+ /** Whether the state has been marked for removing */
+ def isRemoved(): Boolean = {
+ removed
+ }
+
+ /** Whether the state has been been updated */
+ def isUpdated(): Boolean = {
+ updated
+ }
+
+ /**
+ * Internal method to update the state data and reset internal flags in `this`.
+ * This method allows `this` object to be reused across many state records.
+ */
+ def wrap(optionalState: Option[S]): Unit = {
+ optionalState match {
+ case Some(newState) =>
+ this.state = newState
+ defined = true
+
+ case None =>
+ this.state = null.asInstanceOf[S]
+ defined = false
+ }
+ timingOut = false
+ removed = false
+ updated = false
+ }
+
+ /**
+ * Internal method to update the state data and reset internal flags in `this`.
+ * This method allows `this` object to be reused across many state records.
+ */
+ def wrapTiminoutState(newState: S): Unit = {
--- End diff --
Seems to contain a typo? It's also not clear how this is different from wrap, from the comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153294458
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153299111
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44911/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43981921
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+/**
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] and
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream]].
+ * {{{
+ *
+ * }}}
+ */
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise wise it will throw an exception.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value. Note that you cannot update the state if the state is
+ * timing out (that is, `isTimingOut() return true`, or if the state has already been removed by
+ * `remove()`.
+ */
+ def update(newState: S): Unit
+
+ /** Remove the state if it exists. */
+ def remove(): Unit
--- End diff --
BTW I have no strong feeling here other than that it match existing things, if we have them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44484586
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.ClosureCleaner
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * def trackingFunction(data: Option[ValueType], wrappedState: State[StateType]): EmittedType = {
+ * ...
+ * }
+ *
+ * val spec = StateSpec.function(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[KeyType, ValueType, StateType, EmittedDataType] spec =
+ * StateStateSpec.function[KeyType, ValueType, StateType, EmittedDataType](trackingFunction)
+ * .numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[KeyType, ValueType, StateType, EmittedType] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey` */
+ def initialState(rdd: RDD[(KeyType, StateType)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey` */
+ def initialState(javaPairRDD: JavaPairRDD[KeyType, StateType]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used.
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * def trackingFunction(data: Option[ValueType], wrappedState: State[StateType]): EmittedType = {
+ * ...
+ * }
+ *
+ * val spec = StateSpec.function(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[KeyType, ValueType, StateType, EmittedDataType] spec =
+ * StateStateSpec.function[KeyType, ValueType, StateType, EmittedDataType](trackingFunction)
+ * .numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data
+ * @tparam KeyType Class of the keys
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def function[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (Time, KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ ClosureCleaner.clean(trackingFunction, checkSerializable = true)
+ new StateSpecImpl(trackingFunction)
+ }
+
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def function[ValueType, StateType, EmittedType](
+ trackingFunction: (Option[ValueType], State[StateType]) => EmittedType
+ ): StateSpec[Any, ValueType, StateType, EmittedType] = {
--- End diff --
This is a valid point. Somehow the unit test that covers this API did not fail on this. This will be problem. Now if we take `KeyType` as part of the types in `function`, then scala compiler will not be able to infer the key-type (as its not used in the function), and the user will have to explicitly specify the types. This is a little cumbersome.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155629094
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45572/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153295304
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44197538
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
--- End diff --
note to self: Update this to name KeyType, ValueType, etc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44095967
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala ---
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.util
+
+import java.io.{ObjectInputStream, ObjectOutputStream}
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.SparkConf
+import org.apache.spark.streaming.util.OpenHashMapBasedStateMap._
+import org.apache.spark.util.collection.OpenHashMap
+
+/** Internal interface for defining the map that keeps track of sessions. */
+private[streaming] abstract class StateMap[K: ClassTag, S: ClassTag] extends Serializable {
+
+ /** Get the state for a key if it exists */
+ def get(key: K): Option[S]
+
+ /** Get all the keys and states whose updated time is older than the given threshold time */
+ def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)]
+
+ /** Get all the keys and states in this map. */
+ def getAll(): Iterator[(K, S, Long)]
+
+ /** Add or update state */
+ def put(key: K, state: S, updatedTime: Long): Unit
+
+ /** Remove a key */
+ def remove(key: K): Unit
+
+ /**
+ * Shallow copy `this` map to create a new state map.
+ * Updates to the new map should not mutate `this` map.
+ */
+ def copy(): StateMap[K, S]
+
+ def toDebugString(): String = toString()
+}
+
+/** Companion object for [[StateMap]], with utility methods */
+private[streaming] object StateMap {
+ def empty[K: ClassTag, S: ClassTag]: StateMap[K, S] = new EmptyStateMap[K, S]
+
+ def create[K: ClassTag, S: ClassTag](conf: SparkConf): StateMap[K, S] = {
+ val deltaChainThreshold = conf.getInt("spark.streaming.sessionByKey.deltaChainThreshold",
+ DELTA_CHAIN_LENGTH_THRESHOLD)
+ new OpenHashMapBasedStateMap[K, S](64, deltaChainThreshold)
+ }
+}
+
+/** Specific implementation of SessionStore interface representing an empty map */
+private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMap[K, S] {
+ override def put(key: K, session: S, updateTime: Long): Unit = {
+ throw new NotImplementedError("put() should not be called on an EmptyStateMap")
+ }
+ override def get(key: K): Option[S] = None
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = Iterator.empty
+ override def getAll(): Iterator[(K, S, Long)] = Iterator.empty
+ override def copy(): StateMap[K, S] = new EmptyStateMap[K, S]
+ override def remove(key: K): Unit = { }
+ override def toDebugString(): String = ""
+}
+
+
+
+/** Implementation of StateMap based on Spark's OpenHashMap */
+private[streaming] class OpenHashMapBasedStateMap[K: ClassTag, S: ClassTag](
+ @transient @volatile var parentStateMap: StateMap[K, S],
+ initialCapacity: Int = 64,
+ deltaChainThreshold: Int = DELTA_CHAIN_LENGTH_THRESHOLD
+ ) extends StateMap[K, S] { self =>
+
+ def this(initialCapacity: Int, deltaChainThreshold: Int) = this(
+ new EmptyStateMap[K, S],
+ initialCapacity = initialCapacity,
+ deltaChainThreshold = deltaChainThreshold)
+
+ def this(deltaChainThreshold: Int) = this(
+ initialCapacity = 64, deltaChainThreshold = deltaChainThreshold)
+
+ def this() = this(DELTA_CHAIN_LENGTH_THRESHOLD)
+
+ @transient @volatile private var deltaMap =
+ new OpenHashMap[K, StateInfo[S]](initialCapacity)
+
+ /** Get the session data if it exists */
+ override def get(key: K): Option[S] = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ if (!stateInfo.deleted) {
+ Some(stateInfo.data)
+ } else {
+ None
+ }
+ } else {
+ parentStateMap.get(key)
+ }
+ }
+
+ /** Get all the keys and states whose updated time is older than the give threshold time */
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = {
+ val oldStates = parentStateMap.getByTime(threshUpdatedTime).filter { case (key, value, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.flatMap { case (key, stateInfo) =>
+ if (! stateInfo.deleted && stateInfo.updateTime < threshUpdatedTime) {
--- End diff --
nit: extra space
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43090719
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/TrackeStateDStream.scala ---
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import java.io.{IOException, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.util.StateMap
+import org.apache.spark.util.Utils
+
+private[streaming] case class TrackStateRDDRecord[K: ClassTag, S: ClassTag, T: ClassTag](
+ stateMap: StateMap[K, S], emittedRecords: Seq[T])
+
+
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[dstream] var previousSessionRDDPartition: Partition = null
+ private[dstream] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ _sc: SparkContext,
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (K, Option[V], State[S]) => Option[T],
+ currentTime: Long, timeoutThresholdTime: Option[Long]
+ ) extends RDD[TrackStateRDDRecord[K, S, T]](
+ _sc,
+ List(
+ new OneToOneDependency[TrackStateRDDRecord[K, S, T]](prevStateRDD),
+ new OneToOneDependency(partitionedDataRDD))
+ ) {
+
+ @volatile private var doFullScan = false
+
+ require(partitionedDataRDD.partitioner.nonEmpty)
+ require(partitionedDataRDD.partitioner == prevStateRDD.partitioner)
+
+ override val partitioner = prevStateRDD.partitioner
+
+ override def checkpoint(): Unit = {
+ super.checkpoint()
+ doFullScan = true
+ }
+
+ override def compute(
+ partition: Partition, context: TaskContext): Iterator[TrackStateRDDRecord[K, S, T]] = {
+
+ val stateRDDPartition = partition.asInstanceOf[TrackStateRDDPartition]
+ val prevStateRDDIterator = prevStateRDD.iterator(
+ stateRDDPartition.previousSessionRDDPartition, context)
+ val dataIterator = partitionedDataRDD.iterator(
+ stateRDDPartition.partitionedDataRDDPartition, context)
+ if (!prevStateRDDIterator.hasNext) {
+ throw new SparkException(s"Could not find state map in previous state RDD")
+ }
+
+ val newStateMap = prevStateRDDIterator.next().stateMap.copy()
+ val emittedRecords = new ArrayBuffer[T]
+
+ val wrappedState = new StateImpl[S]()
+
+ dataIterator.foreach { case (key, value) =>
+ wrappedState.wrap(newStateMap.get(key))
+ val emittedRecord = trackingFunction(key, Some(value), wrappedState)
+ if (wrappedState.isRemoved) {
+ newStateMap.remove(key)
+ } else if (wrappedState.isUpdated) {
+ newStateMap.put(key, wrappedState.get(), currentTime)
+ }
+ emittedRecords ++= emittedRecord
--- End diff --
It is better not to materialize all the `emittedRecords` in `compute()`, just one by each iteration, from the current code I think it can be achieved, not sure if there's any other concern.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153000299
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44807/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44098629
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala ---
@@ -350,6 +349,18 @@ class PairDStreamFunctions[K, V](self: DStream[(K, V)])
)
}
+ /** TODO: Add scala docs */
+ def trackStateByKey[S: ClassTag, T: ClassTag](
+ spec: TrackStateSpec[K, V, S, T]): EmittedRecordsDStream[K, V, S, T] = {
+ new EmittedRecordsDStreamImpl[K, V, S, T](
+ new TrackStateDStream[K, V, S, T](
+ self,
+ spec.asInstanceOf[TrackStateSpecImpl[K, V, S, T]]
--- End diff --
The DataFrameReader does not seem to have any getters. So I am not sure how to derive inspiration from it.
From offline discussion, the confusion was that the function is not part of the API of `TrackStateSpec` class. And that is because there the function is specified through `TrackStateSpec.apply(func)`. I had designed it that way because the function is the only spec that must be provided, and the rest is optional. So having the `TrackStateSpec.apply()` take the function ensures that its concise (`TrackStateSpec(func)`) and dumb proof (the user has to specify it).
But if this is confusing, then there are two options.
1. Make `function()` a method with which the user can specify the function. So the user could write either `TrackStateSpec.function(func)` or `TrackStateSpec(func)` and the `function` is visible in the API + docs of the TrackStateSpec class.
2. Just leave it as is and make the docs of `TrackStateSpec` class (and also other docs) clearly show the usage using the `TrackStateSpec` object.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44197477
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ *
--- End diff --
note to self: fill up this doc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44213675
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala ---
@@ -351,6 +351,50 @@ class PairDStreamFunctions[K, V](self: DStream[(K, V)])
}
/**
+ * :: Experimental ::
+ * Return a new DStream of data generated by combining the key-value data in `this` stream
+ * with a continuously updated per-key state. The user-provided state tracking function is
+ * applied on each keyed data item along with its corresponding state. The function can choose to
+ * update/remove the state and return a transformed data, which forms the
+ * [[org.apache.spark.streaming.dstream.EmittedRecordsDStream]].
+ *
+ * The specifications of this transformation is made through the
+ * [[org.apache.spark.streaming.StateSpec StateSpec]] class. Besides the tracking function, there
+ * are a number of optional parameters - initial state data, number of partitions, timeouts, etc.
+ * See the [[org.apache.spark.streaming.StateSpec StateSpec spec docs]] for more details.
+ *
+ * Scala example of using `trackStateByKey`:
+ * {{{
+ * def trackingFunction(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ * // Check state exists, accordingly update/remove state and return data to emit
+ * }
+ *
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ *
+ * }}}
+ *
+ * Java example of using `trackStateByKey`:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
--- End diff --
The Java example should only be in the Java API, doesn't make sense to have it here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155688949
All right. I am merging this for now. I have a few more changes, refactorings and more unit tests that I am working on, but that can be done later as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153000317
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44096254
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala ---
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.util
+
+import java.io.{ObjectInputStream, ObjectOutputStream}
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.SparkConf
+import org.apache.spark.streaming.util.OpenHashMapBasedStateMap._
+import org.apache.spark.util.collection.OpenHashMap
+
+/** Internal interface for defining the map that keeps track of sessions. */
+private[streaming] abstract class StateMap[K: ClassTag, S: ClassTag] extends Serializable {
+
+ /** Get the state for a key if it exists */
+ def get(key: K): Option[S]
+
+ /** Get all the keys and states whose updated time is older than the given threshold time */
+ def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)]
+
+ /** Get all the keys and states in this map. */
+ def getAll(): Iterator[(K, S, Long)]
+
+ /** Add or update state */
+ def put(key: K, state: S, updatedTime: Long): Unit
+
+ /** Remove a key */
+ def remove(key: K): Unit
+
+ /**
+ * Shallow copy `this` map to create a new state map.
+ * Updates to the new map should not mutate `this` map.
+ */
+ def copy(): StateMap[K, S]
+
+ def toDebugString(): String = toString()
+}
+
+/** Companion object for [[StateMap]], with utility methods */
+private[streaming] object StateMap {
+ def empty[K: ClassTag, S: ClassTag]: StateMap[K, S] = new EmptyStateMap[K, S]
+
+ def create[K: ClassTag, S: ClassTag](conf: SparkConf): StateMap[K, S] = {
+ val deltaChainThreshold = conf.getInt("spark.streaming.sessionByKey.deltaChainThreshold",
+ DELTA_CHAIN_LENGTH_THRESHOLD)
+ new OpenHashMapBasedStateMap[K, S](64, deltaChainThreshold)
+ }
+}
+
+/** Specific implementation of SessionStore interface representing an empty map */
+private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMap[K, S] {
+ override def put(key: K, session: S, updateTime: Long): Unit = {
+ throw new NotImplementedError("put() should not be called on an EmptyStateMap")
+ }
+ override def get(key: K): Option[S] = None
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = Iterator.empty
+ override def getAll(): Iterator[(K, S, Long)] = Iterator.empty
+ override def copy(): StateMap[K, S] = new EmptyStateMap[K, S]
+ override def remove(key: K): Unit = { }
+ override def toDebugString(): String = ""
+}
+
+
+
+/** Implementation of StateMap based on Spark's OpenHashMap */
+private[streaming] class OpenHashMapBasedStateMap[K: ClassTag, S: ClassTag](
+ @transient @volatile var parentStateMap: StateMap[K, S],
+ initialCapacity: Int = 64,
+ deltaChainThreshold: Int = DELTA_CHAIN_LENGTH_THRESHOLD
+ ) extends StateMap[K, S] { self =>
+
+ def this(initialCapacity: Int, deltaChainThreshold: Int) = this(
+ new EmptyStateMap[K, S],
+ initialCapacity = initialCapacity,
+ deltaChainThreshold = deltaChainThreshold)
+
+ def this(deltaChainThreshold: Int) = this(
+ initialCapacity = 64, deltaChainThreshold = deltaChainThreshold)
+
+ def this() = this(DELTA_CHAIN_LENGTH_THRESHOLD)
+
+ @transient @volatile private var deltaMap =
+ new OpenHashMap[K, StateInfo[S]](initialCapacity)
+
+ /** Get the session data if it exists */
+ override def get(key: K): Option[S] = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ if (!stateInfo.deleted) {
+ Some(stateInfo.data)
+ } else {
+ None
+ }
+ } else {
+ parentStateMap.get(key)
+ }
+ }
+
+ /** Get all the keys and states whose updated time is older than the give threshold time */
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = {
+ val oldStates = parentStateMap.getByTime(threshUpdatedTime).filter { case (key, value, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.flatMap { case (key, stateInfo) =>
--- End diff --
How about replace `flatMap` with `filter + map`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153309413
**[Test build #44913 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44913/consoleFull)** for PR 9256 at commit [`6a75966`](https://github.com/apache/spark/commit/6a75966cf863566008c6c03ad7fe7924e3a25db9).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `sealed abstract class State[S] `\n * `case class TrackStateSpecImpl[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](`\n * `abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](`\n * ` case class StateInfo[S](`\n * ` class LimitMarker(val num: Int) extends Serializable`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153309421
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44913/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44205053
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
--- End diff --
@rxin Can you comment on the builder pattern used by StateSpec? @pwendell Correct me if I am wrong but one point of confusion for him was that there was no reference to the tracking function in the API of the StateSpec class (e.g. no setter method for the function). This is because the function is specified though the StateSpec object's apply/create method. The reason I did it this way was to force the user to specify the function to create a spec, as that is the only must-provide parameter for `trackStateByKey`
Two options to avoid the confusion are
1. Add a setter for the function instead of taking a function through StateSpec.apply(). So instead of `StateSpec(func)` the user will be doing `StateSpec.function(func)`. I dont like the latter, more verbose.
2. Leave it as is and provide better docs (Patrick didnt have the docs to look at, now there is).
@rxin whats your call.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-150708086
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153314481
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44914/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44197569
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ *
+ * @param trackingFunction
+ * @tparam KeyType Class of keys
+ * @tparam ValueType
+ * @tparam StateType
+ * @tparam EmittedType
+ * @return
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
--- End diff --
The constructor the Statespec does not need classtags. I removed last night.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44214008
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data and
+ * @tparam KeyType Class of the keys
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ new StateSpecImpl[KeyType, ValueType, StateType, EmittedType](trackingFunction)
+ }
--- End diff --
IMO don't have apply(), just create()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44245716
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data and
+ * @tparam KeyType Class of the keys
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
--- End diff --
@pwendell @mateiz Any thoughts on these points.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153000296
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44207353
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ * Create a [[org.apache.spark.streaming.StateSpec StateSpec]] for setting all the specifications
+ * `trackStateByKey` operation on a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * @param trackingFunction The function applied on every data item to manage the associated state
+ * and generate the emitted data and
+ * @tparam KeyType Class of the keys
+ * @tparam ValueType Class of the values
+ * @tparam StateType Class of the states data
+ * @tparam EmittedType Class of the emitted data
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
--- End diff --
@rxin @pwendell There are three questions that I would like to discuss regarding this.
1. Value for a key being presented in function as Seq or Option. Seq allows multiple values for a key to be presented together so that the user can sort them in the correct order before updating state. But it means we have to do the grouping. Potentially additional cost. My thought is that there are user-level ways to get around it (like groupByKey) and we can add this later if we need.
Batch time in the function signature. This is based on some good feedback on the doc. The function would like to know what is the current batch time in which it is being processed and accordingly do its own expiry (if the simple timeout logic is not sufficient). That would make the function signature more complex (key: Key, value: Option[Value], state: State[S], batchTime: Time) => Option[T] Currently there is no good workaround other than the user using DStream.transform to embed the batch time in every record. Very inefficient.
Even if we dont include time the signature (K, Option[V], State[S]) => Option[T] is a daunting signature. It might be worthwhilte to include multiple methods - a simplified one like (Option[V], State[S]) => T and a full one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153975680
I took a broad pass on the public API's and left comments throughout, mostly around naming rather than core structure. What would really be helpful for me is filling in the documentation on the key public classes `State`, `StateSpec`, `trackStateByKey`, and `stateSnapshots()` to make sure I fully understand their semantics. I can take a pass again once that is done, but the high level approach seems good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-152998904
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153001027
**[Test build #44808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44808/consoleFull)** for PR 9256 at commit [`6c02f44`](https://github.com/apache/spark/commit/6c02f4479bbc8822382a74f5dab2caf557ad5ae3).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `sealed abstract class State[S] `\n * `case class TrackStateSpecImpl[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](`\n * `abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](`\n * ` case class StateInfo[S](`\n * ` class LimitMarker(val num: Int) extends Serializable`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44205296
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.language.implicitConversions
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Scala example of using `State`:
+ * {{{
+ * def trackStateFunc(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ *
+ * // Check if state exists
+ * if (state.exists) {
+ *
+ * val existingState = wrappedState.get // Get the existing state
+ *
+ * val shouldRemove = ... // Decide whether to remove the state
+ *
+ * if (shouldRemove) {
+ *
+ * wrappedState.remove() // Remove the state
+ *
+ * } else {
+ *
+ * val newState = ...
+ * wrappedState(newState) // Set the new state
+ *
+ * }
+ * } else {
+ *
+ * val initialState = ...
+ * state.update(initialState) // Set the initial state
+ *
+ * }
+ * }
+ *
+ * }}}
+ *
+ * Java example:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
+ */
+@Experimental
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise it will throw `java.util.NoSuchElementException`.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ *
+ * @throws java.util.NoSuchElementException If the state does not exist.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ *
+ * @throws java.lang.IllegalArgumentException If the state has already been removed, or is
+ * going to be removed
+ */
+ def update(newState: S): Unit
+
+ /**
+ * Remove the state if it exists.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ */
+ def remove(): Unit
+
+ /**
+ * Whether the state is timing out and going to be removed by the system after the current batch.
+ * This timeou can occur if timeout duration has been specified in the
+ * [[org.apache.spark.streaming.StateSpec StatSpec]] and the key has not received any new data
+ * for that timeout duration.
+ */
+ def isTimingOut(): Boolean
+
+ /**
+ * Get the state as an [[scala.Option]]. It will be `Some(state)` if it exists, otherwise `None`.
+ */
+ @inline final def getOption(): Option[S] = if (exists) Some(get()) else None
+
+ @inline final override def toString(): String = {
+ getOption.map { _.toString }.getOrElse("<state not set>")
+ }
+}
+
+private[streaming]
+object State {
+ implicit def toOption[S](state: State[S]): Option[S] = state.getOption()
+}
+
+/** Internal implementation of the [[State]] interface */
+private[streaming] class StateImpl[S] extends State[S] {
+
+ private var state: S = null.asInstanceOf[S]
+ private var defined: Boolean = true
+ private var timingOut: Boolean = false
+ private var updated: Boolean = false
+ private var removed: Boolean = false
+
+ // ========= Public API =========
+ def exists(): Boolean = {
--- End diff --
can you add override modifier to everything that's implementing an interface function
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-150709504
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44266/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155551016
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44199924
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/TrackStateRDD.scala ---
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.rdd
+
+import java.io.{IOException, ObjectInputStream, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark.rdd.{MapPartitionsRDD, RDD}
+import org.apache.spark.streaming.{StateImpl, State}
+import org.apache.spark.streaming.util.{EmptyStateMap, StateMap}
+import org.apache.spark.util.Utils
+import org.apache.spark._
+
+private[streaming] case class TrackStateRDDRecord[K, S, T](
+ var stateMap: StateMap[K, S], var emittedRecords: Seq[T]) {
+ /*
+ private def writeObject(outputStream: ObjectOutputStream): Unit = {
+ outputStream.writeObject(stateMap)
+ outputStream.writeInt(emittedRecords.size)
+ val iterator = emittedRecords.iterator
+ while(iterator.hasNext) {
+ outputStream.writeObject(iterator.next)
+ }
+ }
+
+ private def readObject(inputStream: ObjectInputStream): Unit = {
+ stateMap = inputStream.readObject().asInstanceOf[StateMap[K, S]]
+ val numEmittedRecords = inputStream.readInt()
+ val array = new Array[T](numEmittedRecords)
+ var i = 0
+ while(i < numEmittedRecords) {
+ array(i) = inputStream.readObject().asInstanceOf[T]
+ }
+ emittedRecords = array.toSeq
+ }*/
+}
+
+
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[rdd] var previousSessionRDDPartition: Partition = null
+ private[rdd] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+
+/**
+ * RDD storing the keyed-state of trackStateByKey and corresponding emitted records.
+ * Each partition of this RDD has a single record that contains a StateMap storing
+ */
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (K, Option[V], State[S]) => Option[T],
+ currentTime: Long, timeoutThresholdTime: Option[Long]
+ ) extends RDD[TrackStateRDDRecord[K, S, T]](
+ partitionedDataRDD.sparkContext,
+ List(
+ new OneToOneDependency[TrackStateRDDRecord[K, S, T]](prevStateRDD),
+ new OneToOneDependency(partitionedDataRDD))
+ ) {
+
+ @volatile private var doFullScan = false
+
+ require(prevStateRDD.partitioner.nonEmpty)
+ require(partitionedDataRDD.partitioner == prevStateRDD.partitioner)
+
+ override val partitioner = prevStateRDD.partitioner
+
+ override def checkpoint(): Unit = {
+ super.checkpoint()
+ doFullScan = true
+ }
+
+ override def compute(
+ partition: Partition, context: TaskContext): Iterator[TrackStateRDDRecord[K, S, T]] = {
+
+ val stateRDDPartition = partition.asInstanceOf[TrackStateRDDPartition]
+ val prevStateRDDIterator = prevStateRDD.iterator(
+ stateRDDPartition.previousSessionRDDPartition, context)
+ val dataIterator = partitionedDataRDD.iterator(
+ stateRDDPartition.partitionedDataRDDPartition, context)
+
+ val newStateMap = if (prevStateRDDIterator.hasNext) {
+ prevStateRDDIterator.next().stateMap.copy()
+ } else {
+ new EmptyStateMap[K, S]()
+ }
+
+ val emittedRecords = new ArrayBuffer[T]
+
+ val wrappedState = new StateImpl[S]()
+
+ dataIterator.foreach { case (key, value) =>
+ wrappedState.wrap(newStateMap.get(key))
+ val emittedRecord = trackingFunction(key, Some(value), wrappedState)
+ if (wrappedState.isRemoved) {
+ newStateMap.remove(key)
+ } else if (wrappedState.isUpdated) {
+ newStateMap.put(key, wrappedState.get(), currentTime)
+ }
+ emittedRecords ++= emittedRecord
+ }
+
+ if (doFullScan) {
+ if (timeoutThresholdTime.isDefined) {
+ newStateMap.getByTime(timeoutThresholdTime.get).foreach { case (key, state, _) =>
+ wrappedState.wrapTiminoutState(state)
+ val emittedRecord = trackingFunction(key, None, wrappedState)
+ emittedRecords ++= emittedRecord
+ newStateMap.remove(key)
+ }
+ }
+ }
+
+ Iterator(TrackStateRDDRecord(newStateMap, emittedRecords))
+ }
+
+ override protected def getPartitions: Array[Partition] = {
+ Array.tabulate(prevStateRDD.partitions.length) { i =>
+ new TrackStateRDDPartition(i, prevStateRDD, partitionedDataRDD)}
+ }
+
+ override def clearDependencies() {
--- End diff --
that should have been caught by scala style check. We strictly avoid this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153000976
**[Test build #44808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44808/consoleFull)** for PR 9256 at commit [`6c02f44`](https://github.com/apache/spark/commit/6c02f4479bbc8822382a74f5dab2caf557ad5ae3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44213890
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.language.implicitConversions
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Scala example of using `State`:
+ * {{{
+ * def trackStateFunc(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ *
+ * // Check if state exists
+ * if (state.exists) {
+ *
+ * val existingState = wrappedState.get // Get the existing state
+ *
+ * val shouldRemove = ... // Decide whether to remove the state
+ *
+ * if (shouldRemove) {
+ *
+ * wrappedState.remove() // Remove the state
+ *
+ * } else {
+ *
+ * val newState = ...
+ * wrappedState(newState) // Set the new state
+ *
+ * }
+ * } else {
+ *
+ * val initialState = ...
+ * state.update(initialState) // Set the initial state
+ *
+ * }
+ * }
+ *
+ * }}}
+ *
+ * Java example:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
+ */
+@Experimental
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise it will throw `java.util.NoSuchElementException`.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ *
+ * @throws java.util.NoSuchElementException If the state does not exist.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ *
+ * @throws java.lang.IllegalArgumentException If the state has already been removed, or is
+ * going to be removed
+ */
+ def update(newState: S): Unit
+
+ /**
+ * Remove the state if it exists.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ */
+ def remove(): Unit
+
+ /**
+ * Whether the state is timing out and going to be removed by the system after the current batch.
+ * This timeou can occur if timeout duration has been specified in the
+ * [[org.apache.spark.streaming.StateSpec StatSpec]] and the key has not received any new data
+ * for that timeout duration.
+ */
+ def isTimingOut(): Boolean
+
+ /**
+ * Get the state as an [[scala.Option]]. It will be `Some(state)` if it exists, otherwise `None`.
+ */
+ @inline final def getOption(): Option[S] = if (exists) Some(get()) else None
+
+ @inline final override def toString(): String = {
+ getOption.map { _.toString }.getOrElse("<state not set>")
+ }
+}
+
+private[streaming]
+object State {
+ implicit def toOption[S](state: State[S]): Option[S] = state.getOption()
+}
--- End diff --
Why do we need this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153000295
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44233225
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.language.implicitConversions
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Scala example of using `State`:
+ * {{{
+ * def trackStateFunc(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ *
+ * // Check if state exists
+ * if (state.exists) {
+ *
+ * val existingState = wrappedState.get // Get the existing state
+ *
+ * val shouldRemove = ... // Decide whether to remove the state
+ *
+ * if (shouldRemove) {
+ *
+ * wrappedState.remove() // Remove the state
+ *
+ * } else {
+ *
+ * val newState = ...
+ * wrappedState(newState) // Set the new state
+ *
+ * }
+ * } else {
+ *
+ * val initialState = ...
+ * state.update(initialState) // Set the initial state
+ *
+ * }
+ * }
+ *
+ * }}}
+ *
+ * Java example:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
+ */
+@Experimental
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise it will throw `java.util.NoSuchElementException`.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ *
+ * @throws java.util.NoSuchElementException If the state does not exist.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ *
+ * @throws java.lang.IllegalArgumentException If the state has already been removed, or is
+ * going to be removed
+ */
+ def update(newState: S): Unit
+
+ /**
+ * Remove the state if it exists.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ */
+ def remove(): Unit
+
+ /**
+ * Whether the state is timing out and going to be removed by the system after the current batch.
+ * This timeou can occur if timeout duration has been specified in the
+ * [[org.apache.spark.streaming.StateSpec StatSpec]] and the key has not received any new data
+ * for that timeout duration.
+ */
+ def isTimingOut(): Boolean
+
+ /**
+ * Get the state as an [[scala.Option]]. It will be `Some(state)` if it exists, otherwise `None`.
+ */
+ @inline final def getOption(): Option[S] = if (exists) Some(get()) else None
+
+ @inline final override def toString(): String = {
+ getOption.map { _.toString }.getOrElse("<state not set>")
+ }
+}
+
+private[streaming]
+object State {
+ implicit def toOption[S](state: State[S]): Option[S] = state.getOption()
+}
--- End diff --
We dont "need" it. But its just convenient for people to automatically convert to an option, so that they can write `state.getOrElse(...)` in Scala.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44092347
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+/**
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] and
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream]].
+ * {{{
+ *
+ * }}}
+ */
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise wise it will throw an exception.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value. Note that you cannot update the state if the state is
+ * timing out (that is, `isTimingOut() return true`, or if the state has already been removed by
+ * `remove()`.
+ */
+ def update(newState: S): Unit
+
+ /** Remove the state if it exists. */
+ def remove(): Unit
+
+ /** Is the state going to be timed out by the system after this batch interval */
+ def isTimingOut(): Boolean
+
+ @inline final def getOption(): Option[S] = Option(get())
+
+ /** Get the state if it exists, otherwise return the default value */
+ @inline final def getOrElse[S1 >: S](default: => S1): S1 = {
+ if (exists) this.get else default
+ }
+
+ @inline final override def toString(): String = {
+ getOption.map { _.toString }.getOrElse("<state not set>")
+ }
+}
+
+/** Internal implementation of the [[State]] interface */
+private[streaming] class StateImpl[S] extends State[S] {
+
+ private var state: S = null.asInstanceOf[S]
+ private var defined: Boolean = true
+ private var timingOut: Boolean = false
+ private var updated: Boolean = false
+ private var removed: Boolean = false
+
+ // ========= Public API =========
+ def exists(): Boolean = {
+ defined
+ }
+
+ def get(): S = {
+ state
--- End diff --
Good catch. On the todo list to add State unit tests to verify this behavior.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44055168
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+
+abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ ssc: StreamingContext) extends DStream[T](ssc) {
+
+ def stateSnapshots(): DStream[(K, S)]
--- End diff --
snapshotStream?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44087502
--- Diff: examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala ---
@@ -44,18 +44,6 @@ object StatefulNetworkWordCount {
StreamingExamples.setStreamingLogLevels()
- val updateFunc = (values: Seq[Int], state: Option[Int]) => {
--- End diff --
Since `updateStateByKey` isn't removed, maybe it's better to keep this example and create a new file for `trackStateByKey`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44230031
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
--- End diff --
Basically what I'm saying is don't have function on StateSpec class, and only have function on StateSpec object, since you must define a function.
And don't have apply on StateSpec object.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43980502
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala ---
@@ -350,6 +349,18 @@ class PairDStreamFunctions[K, V](self: DStream[(K, V)])
)
}
+ /** TODO: Add scala docs */
+ def trackStateByKey[S: ClassTag, T: ClassTag](
--- End diff --
Especially if there can be a doc that describes (K, V, S, T) and what their semantics are.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by huitseeker <gi...@git.apache.org>.
Github user huitseeker commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44187283
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ *
+ * @param trackingFunction
+ * @tparam KeyType Class of keys
+ * @tparam ValueType
+ * @tparam StateType
+ * @tparam EmittedType
+ * @return
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
--- End diff --
`StateType` `EmittedType` here are not constrained in `apply` and thus have no `ClassTag` available for the constructor.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44214022
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+/**
+ * :: Experimental ::
+ * DStream representing the stream of records emitted after the `trackStateByKey` operation
+ * on a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] in Scala.
+ * Additionally, it also gives access to the stream of state snapshots, that is, the state data of
+ * all keys after a batch has updated them.
+ *
+ * @tparam K Class of the state key
+ * @tparam S Class of the state data
+ * @tparam T Class of the emitted records
+ */
+@Experimental
+sealed abstract class EmittedRecordsDStream[K, S, T: ClassTag](
+ ssc: StreamingContext) extends DStream[T](ssc) {
+
+ /** Return a pair DStream where each RDD is the snapshot of the state of all the keys. */
+ def stateSnapshots(): DStream[(K, S)]
+}
+
+/** Internal implementation of the [[EmittedRecordsDStream]] */
+private[streaming] class EmittedRecordsDStreamImpl[
+ K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackStateDStream: TrackStateDStream[K, V, S, T])
+ extends EmittedRecordsDStream[K, S, T](trackStateDStream.context) {
+
+ override def slideDuration: Duration = trackStateDStream.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(trackStateDStream)
+
+ override def compute(validTime: Time): Option[RDD[T]] = {
+ trackStateDStream.getOrCompute(validTime).map { _.flatMap[T] { _.emittedRecords } }
+ }
+
+ def stateSnapshots(): DStream[(K, S)] = {
+ trackStateDStream.flatMap[(K, S)] {
+ _.stateMap.getAll().map { case (k, s, _) => (k, s) }.toTraversable }
+ }
+}
+
+/**
+ * A DStream that allows per-key state to be maintains, and arbitrary records to be generated
+ * based on updates to the state.
+ *
+ * @param parent Parent (key, value) stream that is the source
+ * @param spec Specifications of the trackStateByKey operation
+ * @tparam K Key type
+ * @tparam V Value type
+ * @tparam S Type of the state maintained
+ * @tparam T Type of the eiitted records
+ */
+private[streaming] class TrackStateDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ parent: DStream[(K, V)], spec: StateSpecImpl[K, V, S, T])
+ extends DStream[TrackStateRDDRecord[K, S, T]](parent.context) {
+
+ persist(StorageLevel.MEMORY_ONLY)
+
+ private val partitioner = spec.getPartitioner().getOrElse(
+ new HashPartitioner(ssc.sc.defaultParallelism))
+
+ private val trackingFunction = spec.getFunction()
+
+ override def slideDuration: Duration = parent.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(parent)
+
+ override val mustCheckpoint = true
+
+ /** Method that generates a RDD for the given time */
+ override def compute(validTime: Time): Option[RDD[TrackStateRDDRecord[K, S, T]]] = {
+ val prevStateRDD = getOrCompute(validTime - slideDuration).getOrElse {
+ TrackStateRDD.createFromPairRDD[K, V, S, T](
+ spec.getInitialStateRDD().getOrElse(new EmptyRDD[(K, S)](ssc.sparkContext)),
+ partitioner,
+ validTime.milliseconds
+ )
+ }
+ val newDataRDD = parent.getOrCompute(validTime).get
+ val partitionedDataRDD = newDataRDD.partitionBy(partitioner)
+ val timeoutThresholdTime = spec.getTimeoutInterval().map { interval =>
+ (validTime - interval).milliseconds
--- End diff --
(And maybe also say what it takes to make it exact, i.e. if your timeout is a multiple of the batch interval).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44205466
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
--- End diff --
I really don't like apply on an object to create everything. I like
```
StateSpec.function(...)
```
as the only constructor, which also makes it obvious it is setting a function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155613148
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153299110
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by huitseeker <gi...@git.apache.org>.
Github user huitseeker commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44190238
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala ---
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.util
+
+import java.io.{ObjectInputStream, ObjectOutputStream}
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.SparkConf
+import org.apache.spark.streaming.util.OpenHashMapBasedStateMap._
+import org.apache.spark.util.collection.OpenHashMap
+
+/** Internal interface for defining the map that keeps track of sessions. */
+private[streaming] abstract class StateMap[K: ClassTag, S: ClassTag] extends Serializable {
+
+ /** Get the state for a key if it exists */
+ def get(key: K): Option[S]
+
+ /** Get all the keys and states whose updated time is older than the given threshold time */
+ def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)]
+
+ /** Get all the keys and states in this map. */
+ def getAll(): Iterator[(K, S, Long)]
+
+ /** Add or update state */
+ def put(key: K, state: S, updatedTime: Long): Unit
+
+ /** Remove a key */
+ def remove(key: K): Unit
+
+ /**
+ * Shallow copy `this` map to create a new state map.
+ * Updates to the new map should not mutate `this` map.
+ */
+ def copy(): StateMap[K, S]
+
+ def toDebugString(): String = toString()
+}
+
+/** Companion object for [[StateMap]], with utility methods */
+private[streaming] object StateMap {
+ def empty[K: ClassTag, S: ClassTag]: StateMap[K, S] = new EmptyStateMap[K, S]
+
+ def create[K: ClassTag, S: ClassTag](conf: SparkConf): StateMap[K, S] = {
+ val deltaChainThreshold = conf.getInt("spark.streaming.sessionByKey.deltaChainThreshold",
+ DELTA_CHAIN_LENGTH_THRESHOLD)
+ new OpenHashMapBasedStateMap[K, S](64, deltaChainThreshold)
+ }
+}
+
+/** Specific implementation of SessionStore interface representing an empty map */
+private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMap[K, S] {
+ override def put(key: K, session: S, updateTime: Long): Unit = {
+ throw new NotImplementedError("put() should not be called on an EmptyStateMap")
+ }
+ override def get(key: K): Option[S] = None
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = Iterator.empty
+ override def getAll(): Iterator[(K, S, Long)] = Iterator.empty
+ override def copy(): StateMap[K, S] = new EmptyStateMap[K, S]
+ override def remove(key: K): Unit = { }
+ override def toDebugString(): String = ""
+}
+
+
+
+/** Implementation of StateMap based on Spark's OpenHashMap */
+private[streaming] class OpenHashMapBasedStateMap[K: ClassTag, S: ClassTag](
+ @transient @volatile var parentStateMap: StateMap[K, S],
+ initialCapacity: Int = 64,
+ deltaChainThreshold: Int = DELTA_CHAIN_LENGTH_THRESHOLD
+ ) extends StateMap[K, S] { self =>
+
+ def this(initialCapacity: Int, deltaChainThreshold: Int) = this(
+ new EmptyStateMap[K, S],
+ initialCapacity = initialCapacity,
+ deltaChainThreshold = deltaChainThreshold)
+
+ def this(deltaChainThreshold: Int) = this(
+ initialCapacity = 64, deltaChainThreshold = deltaChainThreshold)
+
+ def this() = this(DELTA_CHAIN_LENGTH_THRESHOLD)
+
+ @transient @volatile private var deltaMap =
+ new OpenHashMap[K, StateInfo[S]](initialCapacity)
+
+ /** Get the session data if it exists */
+ override def get(key: K): Option[S] = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ if (!stateInfo.deleted) {
+ Some(stateInfo.data)
+ } else {
+ None
+ }
+ } else {
+ parentStateMap.get(key)
+ }
+ }
+
+ /** Get all the keys and states whose updated time is older than the give threshold time */
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = {
+ val oldStates = parentStateMap.getByTime(threshUpdatedTime).filter { case (key, value, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.flatMap { case (key, stateInfo) =>
+ if (! stateInfo.deleted && stateInfo.updateTime < threshUpdatedTime) {
+ Some((key, stateInfo.data, stateInfo.updateTime))
+ } else None
+ }
+ oldStates ++ updatedStates
+ }
+
+ /** Get all the keys and states in this map. */
+ override def getAll(): Iterator[(K, S, Long)] = {
+
+ val oldStates = parentStateMap.getAll().filter { case (key, _, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.filter { ! _._2.deleted }.map { case (key, stateInfo) =>
+ (key, stateInfo.data, stateInfo.updateTime)
+ }
+ oldStates ++ updatedStates
+ }
+
+ /** Add or update state */
+ override def put(key: K, state: S, updateTime: Long): Unit = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ stateInfo.update(state, updateTime)
+ } else {
+ deltaMap.update(key, new StateInfo(state, updateTime))
+ }
+ }
+
+ /** Remove a state */
+ override def remove(key: K): Unit = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ stateInfo.markDeleted()
+ } else {
+ val newInfo = new StateInfo[S](deleted = true)
+ deltaMap.update(key, newInfo)
+ }
+ }
+
+ /**
+ * Shallow copy the map to create a new session store. Updates to the new map
+ * should not mutate `this` map.
+ */
+ override def copy(): StateMap[K, S] = {
+ new OpenHashMapBasedStateMap[K, S](this, deltaChainThreshold = deltaChainThreshold)
+ }
+
+ def shouldCompact: Boolean = {
+ deltaChainLength >= deltaChainThreshold
+ }
+
+ def deltaChainLength: Int = parentStateMap match {
+ case map: OpenHashMapBasedStateMap[_, _] => map.deltaChainLength + 1
+ case _ => 0
+ }
+
+ def approxSize: Int = deltaMap.size + {
--- End diff --
Perhaps scaladoc that this is an overestimation ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155594327
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45550/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44213622
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+/**
+ * :: Experimental ::
+ * DStream representing the stream of records emitted after the `trackStateByKey` operation
+ * on a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] in Scala.
+ * Additionally, it also gives access to the stream of state snapshots, that is, the state data of
+ * all keys after a batch has updated them.
+ *
+ * @tparam K Class of the state key
+ * @tparam S Class of the state data
+ * @tparam T Class of the emitted records
+ */
+@Experimental
+sealed abstract class EmittedRecordsDStream[K, S, T: ClassTag](
--- End diff --
I'd just call it EmittedDStream.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155615006
**[Test build #45572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45572/consoleFull)** for PR 9256 at commit [`ae64786`](https://github.com/apache/spark/commit/ae64786fd937002a2cc1f80518d54e970a6bbb21).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43981556
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+/**
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] and
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream]].
+ * {{{
+ *
+ * }}}
+ */
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise wise it will throw an exception.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value. Note that you cannot update the state if the state is
+ * timing out (that is, `isTimingOut() return true`, or if the state has already been removed by
+ * `remove()`.
+ */
+ def update(newState: S): Unit
+
+ /** Remove the state if it exists. */
+ def remove(): Unit
--- End diff --
Should this be `delete` or `destroy`? Not sure if we have used similar terminology elsewhere. Also it would be good to state the semantics of calling this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43981603
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+/**
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] and
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream]].
+ * {{{
+ *
+ * }}}
+ */
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise wise it will throw an exception.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value. Note that you cannot update the state if the state is
+ * timing out (that is, `isTimingOut() return true`, or if the state has already been removed by
+ * `remove()`.
+ */
+ def update(newState: S): Unit
--- End diff --
minor nit: but prefer `def update(state: S): Unit` since it's implied by the semantics of update that it is new.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44503186
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/TrackStateRDD.scala ---
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.rdd
+
+import java.io.{IOException, ObjectInputStream, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark.rdd.{MapPartitionsRDD, RDD}
+import org.apache.spark.streaming.{Time, StateImpl, State}
+import org.apache.spark.streaming.util.{EmptyStateMap, StateMap}
+import org.apache.spark.util.Utils
+import org.apache.spark._
+
+/**
+ * Record storing the keyed-state [[TrackStateRDD]]. Each record contains a [[StateMap]] and a
+ * sequence of records returned by the tracking function of `trackStateByKey`.
+ */
+private[streaming] case class TrackStateRDDRecord[K, S, T](
+ var stateMap: StateMap[K, S], var emittedRecords: Seq[T])
+
+/**
+ * Partition of the [[TrackStateRDD]], which depends on corresponding partitions of prev state
+ * RDD, and a partitioned keyed-data RDD
+ */
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[rdd] var previousSessionRDDPartition: Partition = null
+ private[rdd] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+
+/**
+ * RDD storing the keyed-state of `trackStateByKey` and corresponding emitted records.
+ * Each partition of this RDD has a single record of type [[TrackStateRDDRecord]]. This contains a
+ * [[StateMap]] (containing the keyed-states) and the sequence of records returned by the tracking
+ * function of `trackStateByKey`.
+ * @param prevStateRDD The previous TrackStateRDD on whose StateMap data `this` RDD will be created
+ * @param partitionedDataRDD The partitioned data RDD which is used update the previous StateMaps
+ * in the `prevStateRDD` to create `this` RDD
+ * @param trackingFunction The function that will be used to update state and return new data
+ * @param batchTime The time of the batch to which this RDD belongs to. Use to update
+ */
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (Time, K, Option[V], State[S]) => Option[T],
+ batchTime: Time, timeoutThresholdTime: Option[Long]
--- End diff --
no doc for `timeoutThresholdTime `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155629092
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44233579
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/TrackStateRDD.scala ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.rdd
+
+import java.io.{IOException, ObjectInputStream, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark.rdd.{MapPartitionsRDD, RDD}
+import org.apache.spark.streaming.{StateImpl, State}
+import org.apache.spark.streaming.util.{EmptyStateMap, StateMap}
+import org.apache.spark.util.Utils
+import org.apache.spark._
+
+private[streaming] case class TrackStateRDDRecord[K, S, T](
+ var stateMap: StateMap[K, S], var emittedRecords: Seq[T]) {
+ /*
+ private def writeObject(outputStream: ObjectOutputStream): Unit = {
+ outputStream.writeObject(stateMap)
+ outputStream.writeInt(emittedRecords.size)
+ val iterator = emittedRecords.iterator
+ while(iterator.hasNext) {
+ outputStream.writeObject(iterator.next)
+ }
+ }
+
+ private def readObject(inputStream: ObjectInputStream): Unit = {
+ stateMap = inputStream.readObject().asInstanceOf[StateMap[K, S]]
+ val numEmittedRecords = inputStream.readInt()
+ val array = new Array[T](numEmittedRecords)
+ var i = 0
+ while(i < numEmittedRecords) {
+ array(i) = inputStream.readObject().asInstanceOf[T]
+ }
+ emittedRecords = array.toSeq
+ }*/
--- End diff --
Still a bit of WIP. I wanted to optimized the serialization a little bit, but I will move this work to a different PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44070212
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala ---
@@ -350,6 +349,18 @@ class PairDStreamFunctions[K, V](self: DStream[(K, V)])
)
}
+ /** TODO: Add scala docs */
+ def trackStateByKey[S: ClassTag, T: ClassTag](
--- End diff --
Actually, `ClassTags` in `TrackStateSpec` and `TrackStateSpecImpl` are unnecessary. We can remove them and keep `S: ClassTag, T: ClassTag` in `trackStateByKey`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153295344
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-154141796
@tdas can you also build an example file so we can look at the example itself?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43982032
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/TrackStateSpec.scala ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{HashPartitioner, Partitioner}
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * Abstract class having all the specifications of DStream.trackStateByKey().
+ * Use the `TrackStateSpec.create()` or `TrackStateSpec.create()` to create instances of this class.
+ *
+ * {{{
+ * TrackStateSpec(trackingFunction) // in Scala
+ * TrackStateSpec.create(trackingFunction) // in Java
+ * }}}
+ */
+sealed abstract class TrackStateSpec[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag]
+ extends Serializable {
+
+ def initialState(rdd: RDD[(K, S)]): this.type
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ def numPartitions(numPartitions: Int): this.type
+ def partitioner(partitioner: Partitioner): this.type
+
+ def timeout(interval: Duration): this.type
--- End diff --
A doc here to precisely define timeouts would be really helpful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153294442
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155550977
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by huitseeker <gi...@git.apache.org>.
Github user huitseeker commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44190145
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala ---
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.util
+
+import java.io.{ObjectInputStream, ObjectOutputStream}
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.SparkConf
+import org.apache.spark.streaming.util.OpenHashMapBasedStateMap._
+import org.apache.spark.util.collection.OpenHashMap
+
+/** Internal interface for defining the map that keeps track of sessions. */
+private[streaming] abstract class StateMap[K: ClassTag, S: ClassTag] extends Serializable {
+
+ /** Get the state for a key if it exists */
+ def get(key: K): Option[S]
+
+ /** Get all the keys and states whose updated time is older than the given threshold time */
+ def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)]
+
+ /** Get all the keys and states in this map. */
+ def getAll(): Iterator[(K, S, Long)]
+
+ /** Add or update state */
+ def put(key: K, state: S, updatedTime: Long): Unit
+
+ /** Remove a key */
+ def remove(key: K): Unit
+
+ /**
+ * Shallow copy `this` map to create a new state map.
+ * Updates to the new map should not mutate `this` map.
+ */
+ def copy(): StateMap[K, S]
+
+ def toDebugString(): String = toString()
+}
+
+/** Companion object for [[StateMap]], with utility methods */
+private[streaming] object StateMap {
+ def empty[K: ClassTag, S: ClassTag]: StateMap[K, S] = new EmptyStateMap[K, S]
+
+ def create[K: ClassTag, S: ClassTag](conf: SparkConf): StateMap[K, S] = {
+ val deltaChainThreshold = conf.getInt("spark.streaming.sessionByKey.deltaChainThreshold",
+ DELTA_CHAIN_LENGTH_THRESHOLD)
+ new OpenHashMapBasedStateMap[K, S](64, deltaChainThreshold)
+ }
+}
+
+/** Specific implementation of SessionStore interface representing an empty map */
+private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMap[K, S] {
+ override def put(key: K, session: S, updateTime: Long): Unit = {
+ throw new NotImplementedError("put() should not be called on an EmptyStateMap")
+ }
+ override def get(key: K): Option[S] = None
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = Iterator.empty
+ override def getAll(): Iterator[(K, S, Long)] = Iterator.empty
+ override def copy(): StateMap[K, S] = new EmptyStateMap[K, S]
+ override def remove(key: K): Unit = { }
+ override def toDebugString(): String = ""
+}
+
+
+
+/** Implementation of StateMap based on Spark's OpenHashMap */
+private[streaming] class OpenHashMapBasedStateMap[K: ClassTag, S: ClassTag](
+ @transient @volatile var parentStateMap: StateMap[K, S],
+ initialCapacity: Int = 64,
+ deltaChainThreshold: Int = DELTA_CHAIN_LENGTH_THRESHOLD
+ ) extends StateMap[K, S] { self =>
+
+ def this(initialCapacity: Int, deltaChainThreshold: Int) = this(
+ new EmptyStateMap[K, S],
+ initialCapacity = initialCapacity,
+ deltaChainThreshold = deltaChainThreshold)
+
+ def this(deltaChainThreshold: Int) = this(
+ initialCapacity = 64, deltaChainThreshold = deltaChainThreshold)
+
+ def this() = this(DELTA_CHAIN_LENGTH_THRESHOLD)
+
+ @transient @volatile private var deltaMap =
+ new OpenHashMap[K, StateInfo[S]](initialCapacity)
+
+ /** Get the session data if it exists */
+ override def get(key: K): Option[S] = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ if (!stateInfo.deleted) {
+ Some(stateInfo.data)
+ } else {
+ None
+ }
+ } else {
+ parentStateMap.get(key)
+ }
+ }
+
+ /** Get all the keys and states whose updated time is older than the give threshold time */
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = {
+ val oldStates = parentStateMap.getByTime(threshUpdatedTime).filter { case (key, value, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.flatMap { case (key, stateInfo) =>
+ if (! stateInfo.deleted && stateInfo.updateTime < threshUpdatedTime) {
+ Some((key, stateInfo.data, stateInfo.updateTime))
+ } else None
+ }
+ oldStates ++ updatedStates
+ }
+
+ /** Get all the keys and states in this map. */
+ override def getAll(): Iterator[(K, S, Long)] = {
+
+ val oldStates = parentStateMap.getAll().filter { case (key, _, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.filter { ! _._2.deleted }.map { case (key, stateInfo) =>
+ (key, stateInfo.data, stateInfo.updateTime)
+ }
+ oldStates ++ updatedStates
+ }
+
+ /** Add or update state */
+ override def put(key: K, state: S, updateTime: Long): Unit = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ stateInfo.update(state, updateTime)
+ } else {
+ deltaMap.update(key, new StateInfo(state, updateTime))
+ }
+ }
+
+ /** Remove a state */
+ override def remove(key: K): Unit = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ stateInfo.markDeleted()
+ } else {
+ val newInfo = new StateInfo[S](deleted = true)
+ deltaMap.update(key, newInfo)
+ }
+ }
+
+ /**
+ * Shallow copy the map to create a new session store. Updates to the new map
+ * should not mutate `this` map.
+ */
+ override def copy(): StateMap[K, S] = {
+ new OpenHashMapBasedStateMap[K, S](this, deltaChainThreshold = deltaChainThreshold)
+ }
+
+ def shouldCompact: Boolean = {
+ deltaChainLength >= deltaChainThreshold
--- End diff --
I'm curious about benchmarking having `getAll`, (possibly a trigger of `doFullScan` in `TrackStateRDD` ?) trigger this would help.
There's a bit of research by the authors of [this paper](https://www.lri.fr/~conchon/publis/spds-esop08.pdf) on the benefits of *frequent* updates of such 'path compression'.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44214016
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+/**
+ * :: Experimental ::
+ * DStream representing the stream of records emitted after the `trackStateByKey` operation
+ * on a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] in Scala.
+ * Additionally, it also gives access to the stream of state snapshots, that is, the state data of
+ * all keys after a batch has updated them.
+ *
+ * @tparam K Class of the state key
+ * @tparam S Class of the state data
+ * @tparam T Class of the emitted records
+ */
+@Experimental
+sealed abstract class EmittedRecordsDStream[K, S, T: ClassTag](
+ ssc: StreamingContext) extends DStream[T](ssc) {
+
+ /** Return a pair DStream where each RDD is the snapshot of the state of all the keys. */
+ def stateSnapshots(): DStream[(K, S)]
+}
+
+/** Internal implementation of the [[EmittedRecordsDStream]] */
+private[streaming] class EmittedRecordsDStreamImpl[
+ K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackStateDStream: TrackStateDStream[K, V, S, T])
+ extends EmittedRecordsDStream[K, S, T](trackStateDStream.context) {
+
+ override def slideDuration: Duration = trackStateDStream.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(trackStateDStream)
+
+ override def compute(validTime: Time): Option[RDD[T]] = {
+ trackStateDStream.getOrCompute(validTime).map { _.flatMap[T] { _.emittedRecords } }
+ }
+
+ def stateSnapshots(): DStream[(K, S)] = {
+ trackStateDStream.flatMap[(K, S)] {
+ _.stateMap.getAll().map { case (k, s, _) => (k, s) }.toTraversable }
+ }
+}
+
+/**
+ * A DStream that allows per-key state to be maintains, and arbitrary records to be generated
+ * based on updates to the state.
+ *
+ * @param parent Parent (key, value) stream that is the source
+ * @param spec Specifications of the trackStateByKey operation
+ * @tparam K Key type
+ * @tparam V Value type
+ * @tparam S Type of the state maintained
+ * @tparam T Type of the eiitted records
+ */
+private[streaming] class TrackStateDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ parent: DStream[(K, V)], spec: StateSpecImpl[K, V, S, T])
+ extends DStream[TrackStateRDDRecord[K, S, T]](parent.context) {
+
+ persist(StorageLevel.MEMORY_ONLY)
+
+ private val partitioner = spec.getPartitioner().getOrElse(
+ new HashPartitioner(ssc.sc.defaultParallelism))
+
+ private val trackingFunction = spec.getFunction()
+
+ override def slideDuration: Duration = parent.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(parent)
+
+ override val mustCheckpoint = true
+
+ /** Method that generates a RDD for the given time */
+ override def compute(validTime: Time): Option[RDD[TrackStateRDDRecord[K, S, T]]] = {
+ val prevStateRDD = getOrCompute(validTime - slideDuration).getOrElse {
+ TrackStateRDD.createFromPairRDD[K, V, S, T](
+ spec.getInitialStateRDD().getOrElse(new EmptyRDD[(K, S)](ssc.sparkContext)),
+ partitioner,
+ validTime.milliseconds
+ )
+ }
+ val newDataRDD = parent.getOrCompute(validTime).get
+ val partitionedDataRDD = newDataRDD.partitionBy(partitioner)
+ val timeoutThresholdTime = spec.getTimeoutInterval().map { interval =>
+ (validTime - interval).milliseconds
--- End diff --
You should definitely say that the timeout isn't exact in the docs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155437981
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-150708065
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43229671
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/TrackeStateDStream.scala ---
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import java.io.{IOException, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.util.StateMap
+import org.apache.spark.util.Utils
+
+private[streaming] case class TrackStateRDDRecord[K: ClassTag, S: ClassTag, T: ClassTag](
+ stateMap: StateMap[K, S], emittedRecords: Seq[T])
+
+
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[dstream] var previousSessionRDDPartition: Partition = null
+ private[dstream] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ _sc: SparkContext,
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (K, Option[V], State[S]) => Option[T],
+ currentTime: Long, timeoutThresholdTime: Option[Long]
+ ) extends RDD[TrackStateRDDRecord[K, S, T]](
+ _sc,
+ List(
+ new OneToOneDependency[TrackStateRDDRecord[K, S, T]](prevStateRDD),
+ new OneToOneDependency(partitionedDataRDD))
+ ) {
+
+ @volatile private var doFullScan = false
+
+ require(partitionedDataRDD.partitioner.nonEmpty)
+ require(partitionedDataRDD.partitioner == prevStateRDD.partitioner)
+
+ override val partitioner = prevStateRDD.partitioner
+
+ override def checkpoint(): Unit = {
+ super.checkpoint()
+ doFullScan = true
+ }
+
+ override def compute(
+ partition: Partition, context: TaskContext): Iterator[TrackStateRDDRecord[K, S, T]] = {
+
+ val stateRDDPartition = partition.asInstanceOf[TrackStateRDDPartition]
+ val prevStateRDDIterator = prevStateRDD.iterator(
+ stateRDDPartition.previousSessionRDDPartition, context)
+ val dataIterator = partitionedDataRDD.iterator(
+ stateRDDPartition.partitionedDataRDDPartition, context)
+ if (!prevStateRDDIterator.hasNext) {
+ throw new SparkException(s"Could not find state map in previous state RDD")
+ }
+
+ val newStateMap = prevStateRDDIterator.next().stateMap.copy()
+ val emittedRecords = new ArrayBuffer[T]
+
+ val wrappedState = new StateImpl[S]()
+
+ dataIterator.foreach { case (key, value) =>
+ wrappedState.wrap(newStateMap.get(key))
+ val emittedRecord = trackingFunction(key, Some(value), wrappedState)
--- End diff --
Since we defined the tracking function as `(K, Option[V], State[S]) => Option[T]`, so user will expect `None` rather than `Some(null)` if the input value is null, that is a little misleading and will possibly lead to NPE if used like `v.map(_.toString).getOrElse(...)`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44206972
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
--- End diff --
Actually, I correct myself. have `object StateSpec { def apply() ... }` and `class StateSpec { def function(...) }` makes the syntax cumbersome as that can be called only as `StateSpec().function(func)` ( not `StateSpec.function(func)` that I had thought earlier). To achieve `StateSpec.function(func)`, the method `function` must be a method of the object, like `StateSpec.apply()`. That does not solve @pwendell 's problem of not having the setter method `function` in the class.
We could have both - the method `function` on both the class and object. That would solve both problems - use the object `StateSpec.function(func)` and `function` setter in the `StateSpec` class. But that would also allow the user to write `StateSpec.function(func1).function(func2)`. What should the semantics be in that case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155440091
**[Test build #45527 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45527/consoleFull)** for PR 9256 at commit [`fb5a296`](https://github.com/apache/spark/commit/fb5a296ac2c82eec16f2449267f94e98a46c54be).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44205149
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.language.implicitConversions
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Scala example of using `State`:
+ * {{{
+ * def trackStateFunc(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ *
+ * // Check if state exists
+ * if (state.exists) {
+ *
+ * val existingState = wrappedState.get // Get the existing state
+ *
+ * val shouldRemove = ... // Decide whether to remove the state
+ *
+ * if (shouldRemove) {
+ *
--- End diff --
can we remove some blank lines ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43231944
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/TrackeStateDStream.scala ---
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import java.io.{IOException, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.util.StateMap
+import org.apache.spark.util.Utils
+
+private[streaming] case class TrackStateRDDRecord[K: ClassTag, S: ClassTag, T: ClassTag](
+ stateMap: StateMap[K, S], emittedRecords: Seq[T])
+
+
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[dstream] var previousSessionRDDPartition: Partition = null
+ private[dstream] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ _sc: SparkContext,
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (K, Option[V], State[S]) => Option[T],
+ currentTime: Long, timeoutThresholdTime: Option[Long]
+ ) extends RDD[TrackStateRDDRecord[K, S, T]](
+ _sc,
+ List(
+ new OneToOneDependency[TrackStateRDDRecord[K, S, T]](prevStateRDD),
+ new OneToOneDependency(partitionedDataRDD))
+ ) {
+
+ @volatile private var doFullScan = false
+
+ require(partitionedDataRDD.partitioner.nonEmpty)
+ require(partitionedDataRDD.partitioner == prevStateRDD.partitioner)
+
+ override val partitioner = prevStateRDD.partitioner
+
+ override def checkpoint(): Unit = {
+ super.checkpoint()
+ doFullScan = true
+ }
+
+ override def compute(
+ partition: Partition, context: TaskContext): Iterator[TrackStateRDDRecord[K, S, T]] = {
+
+ val stateRDDPartition = partition.asInstanceOf[TrackStateRDDPartition]
+ val prevStateRDDIterator = prevStateRDD.iterator(
+ stateRDDPartition.previousSessionRDDPartition, context)
+ val dataIterator = partitionedDataRDD.iterator(
+ stateRDDPartition.partitionedDataRDDPartition, context)
+ if (!prevStateRDDIterator.hasNext) {
+ throw new SparkException(s"Could not find state map in previous state RDD")
+ }
+
+ val newStateMap = prevStateRDDIterator.next().stateMap.copy()
+ val emittedRecords = new ArrayBuffer[T]
+
+ val wrappedState = new StateImpl[S]()
+
+ dataIterator.foreach { case (key, value) =>
+ wrappedState.wrap(newStateMap.get(key))
+ val emittedRecord = trackingFunction(key, Some(value), wrappedState)
+ if (wrappedState.isRemoved) {
+ newStateMap.remove(key)
+ } else if (wrappedState.isUpdated) {
+ newStateMap.put(key, wrappedState.get(), currentTime)
+ }
+ emittedRecords ++= emittedRecord
--- End diff --
The comparison to `HadoopRDD` does not apply here. When cached, the records of `HadoopRDD`'s partitions is grabbed through the iterator, unrolled and stored in MemoryStore. There each record is a (key, value) pair, like (String, String). So the unrolled data is stored in the MemoryStore as a ArrayBuffer[(String, String)] (or equivalent), which can be iterated over any number of times.
In our case, the record type is (StateMap, seq-of-emitted-records), which is not the same as a simple (string, string) record of a `HadoopRDD`s. So if that is persisted, it will be store in MemoryStore a ArrayBuffer[(StateMap, seq-of-emitted-records)]. Now if this sequence of emitted records is an iterator, then it will be ArrayBuffer[(StateMap, Iterator[T]]. You can access the record of tuple (map, iterator) multiple times, but you can unroll the iterator *inside* the record **only once**. Subsequent accesses of the ArrayBuffer will lead to a (StateMap, empty-iterator), which violates the RDD principle.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44214064
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/TrackStateRDD.scala ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.rdd
+
+import java.io.{IOException, ObjectInputStream, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark.rdd.{MapPartitionsRDD, RDD}
+import org.apache.spark.streaming.{StateImpl, State}
+import org.apache.spark.streaming.util.{EmptyStateMap, StateMap}
+import org.apache.spark.util.Utils
+import org.apache.spark._
+
+private[streaming] case class TrackStateRDDRecord[K, S, T](
+ var stateMap: StateMap[K, S], var emittedRecords: Seq[T]) {
+ /*
+ private def writeObject(outputStream: ObjectOutputStream): Unit = {
+ outputStream.writeObject(stateMap)
+ outputStream.writeInt(emittedRecords.size)
+ val iterator = emittedRecords.iterator
+ while(iterator.hasNext) {
+ outputStream.writeObject(iterator.next)
+ }
+ }
+
+ private def readObject(inputStream: ObjectInputStream): Unit = {
+ stateMap = inputStream.readObject().asInstanceOf[StateMap[K, S]]
+ val numEmittedRecords = inputStream.readInt()
+ val array = new Array[T](numEmittedRecords)
+ var i = 0
+ while(i < numEmittedRecords) {
+ array(i) = inputStream.readObject().asInstanceOf[T]
+ }
+ emittedRecords = array.toSeq
+ }*/
--- End diff --
Do we need these?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44205157
--- Diff: examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala ---
@@ -44,18 +44,6 @@ object StatefulNetworkWordCount {
StreamingExamples.setStreamingLogLevels()
- val updateFunc = (values: Seq[Int], state: Option[Int]) => {
--- End diff --
if we want to encourage people to move to the new API, it might be ok to not have the old one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-150709502
**[Test build #44266 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44266/consoleFull)** for PR 9256 at commit [`bd9cd94`](https://github.com/apache/spark/commit/bd9cd9449a15914b410a4482d610ce9355f28b54).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class DefaultSource extends HadoopFsRelationProvider with DataSourceRegister `\n * `class TextOutputWriter(path: String, dataSchema: StructType, context: TaskAttemptContext)`\n * `sealed abstract class State[S] `\n * `case class TrackStateSpecImpl[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](`\n * ` case class StateInfo[S](`\n * ` class LimitMarker(val num: Int) extends Serializable`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153297943
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43982459
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala ---
@@ -350,6 +349,18 @@ class PairDStreamFunctions[K, V](self: DStream[(K, V)])
)
}
+ /** TODO: Add scala docs */
+ def trackStateByKey[S: ClassTag, T: ClassTag](
+ spec: TrackStateSpec[K, V, S, T]): EmittedRecordsDStream[K, V, S, T] = {
+ new EmittedRecordsDStreamImpl[K, V, S, T](
+ new TrackStateDStream[K, V, S, T](
+ self,
+ spec.asInstanceOf[TrackStateSpecImpl[K, V, S, T]]
--- End diff --
Seems like this is what we did with DataFrameReader which follows a similar pattern:
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L47
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43095474
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/TrackeStateDStream.scala ---
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import java.io.{IOException, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.util.StateMap
+import org.apache.spark.util.Utils
+
+private[streaming] case class TrackStateRDDRecord[K: ClassTag, S: ClassTag, T: ClassTag](
+ stateMap: StateMap[K, S], emittedRecords: Seq[T])
+
+
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[dstream] var previousSessionRDDPartition: Partition = null
+ private[dstream] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ _sc: SparkContext,
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (K, Option[V], State[S]) => Option[T],
+ currentTime: Long, timeoutThresholdTime: Option[Long]
+ ) extends RDD[TrackStateRDDRecord[K, S, T]](
+ _sc,
+ List(
+ new OneToOneDependency[TrackStateRDDRecord[K, S, T]](prevStateRDD),
+ new OneToOneDependency(partitionedDataRDD))
+ ) {
+
+ @volatile private var doFullScan = false
+
+ require(partitionedDataRDD.partitioner.nonEmpty)
+ require(partitionedDataRDD.partitioner == prevStateRDD.partitioner)
+
+ override val partitioner = prevStateRDD.partitioner
+
+ override def checkpoint(): Unit = {
+ super.checkpoint()
+ doFullScan = true
+ }
+
+ override def compute(
+ partition: Partition, context: TaskContext): Iterator[TrackStateRDDRecord[K, S, T]] = {
+
+ val stateRDDPartition = partition.asInstanceOf[TrackStateRDDPartition]
+ val prevStateRDDIterator = prevStateRDD.iterator(
+ stateRDDPartition.previousSessionRDDPartition, context)
+ val dataIterator = partitionedDataRDD.iterator(
+ stateRDDPartition.partitionedDataRDDPartition, context)
+ if (!prevStateRDDIterator.hasNext) {
+ throw new SparkException(s"Could not find state map in previous state RDD")
+ }
+
+ val newStateMap = prevStateRDDIterator.next().stateMap.copy()
+ val emittedRecords = new ArrayBuffer[T]
+
+ val wrappedState = new StateImpl[S]()
+
+ dataIterator.foreach { case (key, value) =>
+ wrappedState.wrap(newStateMap.get(key))
+ val emittedRecord = trackingFunction(key, Some(value), wrappedState)
--- End diff --
If the parentDStream[(K, V)] generates data such that the corresponding value is null, then the system should respect that null and pass it on. So odd as it may sounds, I think it is right to call the track state function with `Some(null)`.
But you do raise a good point. I will get more feedback on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43982348
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala ---
@@ -350,6 +349,18 @@ class PairDStreamFunctions[K, V](self: DStream[(K, V)])
)
}
+ /** TODO: Add scala docs */
+ def trackStateByKey[S: ClassTag, T: ClassTag](
+ spec: TrackStateSpec[K, V, S, T]): EmittedRecordsDStream[K, V, S, T] = {
+ new EmittedRecordsDStreamImpl[K, V, S, T](
+ new TrackStateDStream[K, V, S, T](
+ self,
+ spec.asInstanceOf[TrackStateSpecImpl[K, V, S, T]]
--- End diff --
This cast is a little weird. Can you just have a single class `TrackStateSpec` that has both getters and setters?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155572332
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44214006
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
--- End diff --
Why do we want StateSpec() to return anything at all? Why not `new StateSpec().function(...)`? It seems more similar to other builder code. Together with the StateSpec.create() methods for common cases, it should be okay, and there's no need for apply() anywhere.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153307960
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44045397
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+
+abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
--- End diff --
What exactly does this do? EmittedDStream seems unclear.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11290][STREAMING] Basic implementation ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9256
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155553721
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153307911
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43089727
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/TrackeStateDStream.scala ---
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import java.io.{IOException, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.util.StateMap
+import org.apache.spark.util.Utils
+
+private[streaming] case class TrackStateRDDRecord[K: ClassTag, S: ClassTag, T: ClassTag](
+ stateMap: StateMap[K, S], emittedRecords: Seq[T])
+
+
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[dstream] var previousSessionRDDPartition: Partition = null
+ private[dstream] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ _sc: SparkContext,
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (K, Option[V], State[S]) => Option[T],
+ currentTime: Long, timeoutThresholdTime: Option[Long]
+ ) extends RDD[TrackStateRDDRecord[K, S, T]](
+ _sc,
+ List(
+ new OneToOneDependency[TrackStateRDDRecord[K, S, T]](prevStateRDD),
+ new OneToOneDependency(partitionedDataRDD))
+ ) {
+
+ @volatile private var doFullScan = false
+
+ require(partitionedDataRDD.partitioner.nonEmpty)
+ require(partitionedDataRDD.partitioner == prevStateRDD.partitioner)
+
+ override val partitioner = prevStateRDD.partitioner
+
+ override def checkpoint(): Unit = {
+ super.checkpoint()
+ doFullScan = true
+ }
+
+ override def compute(
+ partition: Partition, context: TaskContext): Iterator[TrackStateRDDRecord[K, S, T]] = {
+
+ val stateRDDPartition = partition.asInstanceOf[TrackStateRDDPartition]
+ val prevStateRDDIterator = prevStateRDD.iterator(
+ stateRDDPartition.previousSessionRDDPartition, context)
+ val dataIterator = partitionedDataRDD.iterator(
+ stateRDDPartition.partitionedDataRDDPartition, context)
+ if (!prevStateRDDIterator.hasNext) {
+ throw new SparkException(s"Could not find state map in previous state RDD")
+ }
+
+ val newStateMap = prevStateRDDIterator.next().stateMap.copy()
+ val emittedRecords = new ArrayBuffer[T]
+
+ val wrappedState = new StateImpl[S]()
+
+ dataIterator.foreach { case (key, value) =>
+ wrappedState.wrap(newStateMap.get(key))
+ val emittedRecord = trackingFunction(key, Some(value), wrappedState)
--- End diff --
Is it possible `value` will be `null`, would it be better to use `Option(value)`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44246950
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+/**
+ * :: Experimental ::
+ * DStream representing the stream of records emitted after the `trackStateByKey` operation
+ * on a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] in Scala.
+ * Additionally, it also gives access to the stream of state snapshots, that is, the state data of
+ * all keys after a batch has updated them.
+ *
+ * @tparam K Class of the state key
+ * @tparam S Class of the state data
+ * @tparam T Class of the emitted records
+ */
+@Experimental
+sealed abstract class EmittedRecordsDStream[K, S, T: ClassTag](
+ ssc: StreamingContext) extends DStream[T](ssc) {
+
+ /** Return a pair DStream where each RDD is the snapshot of the state of all the keys. */
+ def stateSnapshots(): DStream[(K, S)]
+}
+
+/** Internal implementation of the [[EmittedRecordsDStream]] */
+private[streaming] class EmittedRecordsDStreamImpl[
+ K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackStateDStream: TrackStateDStream[K, V, S, T])
+ extends EmittedRecordsDStream[K, S, T](trackStateDStream.context) {
+
+ override def slideDuration: Duration = trackStateDStream.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(trackStateDStream)
+
+ override def compute(validTime: Time): Option[RDD[T]] = {
+ trackStateDStream.getOrCompute(validTime).map { _.flatMap[T] { _.emittedRecords } }
+ }
+
+ def stateSnapshots(): DStream[(K, S)] = {
+ trackStateDStream.flatMap[(K, S)] {
+ _.stateMap.getAll().map { case (k, s, _) => (k, s) }.toTraversable }
+ }
+}
+
+/**
+ * A DStream that allows per-key state to be maintains, and arbitrary records to be generated
+ * based on updates to the state.
+ *
+ * @param parent Parent (key, value) stream that is the source
+ * @param spec Specifications of the trackStateByKey operation
+ * @tparam K Key type
+ * @tparam V Value type
+ * @tparam S Type of the state maintained
+ * @tparam T Type of the eiitted records
+ */
+private[streaming] class TrackStateDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ parent: DStream[(K, V)], spec: StateSpecImpl[K, V, S, T])
+ extends DStream[TrackStateRDDRecord[K, S, T]](parent.context) {
+
+ persist(StorageLevel.MEMORY_ONLY)
+
+ private val partitioner = spec.getPartitioner().getOrElse(
+ new HashPartitioner(ssc.sc.defaultParallelism))
+
+ private val trackingFunction = spec.getFunction()
+
+ override def slideDuration: Duration = parent.slideDuration
+
+ override def dependencies: List[DStream[_]] = List(parent)
+
+ override val mustCheckpoint = true
+
+ /** Method that generates a RDD for the given time */
+ override def compute(validTime: Time): Option[RDD[TrackStateRDDRecord[K, S, T]]] = {
+ val prevStateRDD = getOrCompute(validTime - slideDuration).getOrElse {
+ TrackStateRDD.createFromPairRDD[K, V, S, T](
+ spec.getInitialStateRDD().getOrElse(new EmptyRDD[(K, S)](ssc.sparkContext)),
+ partitioner,
+ validTime.milliseconds
+ )
+ }
+ val newDataRDD = parent.getOrCompute(validTime).get
+ val partitionedDataRDD = newDataRDD.partitionBy(partitioner)
+ val timeoutThresholdTime = spec.getTimeoutInterval().map { interval =>
+ (validTime - interval).milliseconds
--- End diff --
Actually, in the current implementation, there is no way to make it exact. Since we dont maintain an timestamp-index in the data structure yet, we need to do full scans to find timedout records. And we do full scans infrequently, only at the timeof checkpointing.
Nonetheless, i will document that the exact time of timing out isnt guaranteed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44205421
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/rdd/TrackStateRDD.scala ---
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.rdd
+
+import java.io.{IOException, ObjectInputStream, ObjectOutputStream}
+
+import scala.collection.mutable.ArrayBuffer
+import scala.reflect.ClassTag
+
+import org.apache.spark.rdd.{MapPartitionsRDD, RDD}
+import org.apache.spark.streaming.{StateImpl, State}
+import org.apache.spark.streaming.util.{EmptyStateMap, StateMap}
+import org.apache.spark.util.Utils
+import org.apache.spark._
+
+private[streaming] case class TrackStateRDDRecord[K, S, T](
+ var stateMap: StateMap[K, S], var emittedRecords: Seq[T]) {
+ /*
+ private def writeObject(outputStream: ObjectOutputStream): Unit = {
+ outputStream.writeObject(stateMap)
+ outputStream.writeInt(emittedRecords.size)
+ val iterator = emittedRecords.iterator
+ while(iterator.hasNext) {
+ outputStream.writeObject(iterator.next)
+ }
+ }
+
+ private def readObject(inputStream: ObjectInputStream): Unit = {
+ stateMap = inputStream.readObject().asInstanceOf[StateMap[K, S]]
+ val numEmittedRecords = inputStream.readInt()
+ val array = new Array[T](numEmittedRecords)
+ var i = 0
+ while(i < numEmittedRecords) {
+ array(i) = inputStream.readObject().asInstanceOf[T]
+ }
+ emittedRecords = array.toSeq
+ }*/
+}
+
+
+private[streaming] class TrackStateRDDPartition(
+ idx: Int,
+ @transient private var prevStateRDD: RDD[_],
+ @transient private var partitionedDataRDD: RDD[_]) extends Partition {
+
+ private[rdd] var previousSessionRDDPartition: Partition = null
+ private[rdd] var partitionedDataRDDPartition: Partition = null
+
+ override def index: Int = idx
+ override def hashCode(): Int = idx
+
+ @throws(classOf[IOException])
+ private def writeObject(oos: ObjectOutputStream): Unit = Utils.tryOrIOException {
+ // Update the reference to parent split at the time of task serialization
+ previousSessionRDDPartition = prevStateRDD.partitions(index)
+ partitionedDataRDDPartition = partitionedDataRDD.partitions(index)
+ oos.defaultWriteObject()
+ }
+}
+
+
+/**
+ * RDD storing the keyed-state of trackStateByKey and corresponding emitted records.
+ * Each partition of this RDD has a single record that contains a StateMap storing
+ */
+private[streaming] class TrackStateRDD[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ private var prevStateRDD: RDD[TrackStateRDDRecord[K, S, T]],
+ private var partitionedDataRDD: RDD[(K, V)],
+ trackingFunction: (K, Option[V], State[S]) => Option[T],
+ currentTime: Long, timeoutThresholdTime: Option[Long]
+ ) extends RDD[TrackStateRDDRecord[K, S, T]](
+ partitionedDataRDD.sparkContext,
+ List(
+ new OneToOneDependency[TrackStateRDDRecord[K, S, T]](prevStateRDD),
+ new OneToOneDependency(partitionedDataRDD))
+ ) {
+
+ @volatile private var doFullScan = false
+
+ require(prevStateRDD.partitioner.nonEmpty)
+ require(partitionedDataRDD.partitioner == prevStateRDD.partitioner)
+
+ override val partitioner = prevStateRDD.partitioner
+
+ override def checkpoint(): Unit = {
+ super.checkpoint()
+ doFullScan = true
+ }
+
+ override def compute(
+ partition: Partition, context: TaskContext): Iterator[TrackStateRDDRecord[K, S, T]] = {
+
+ val stateRDDPartition = partition.asInstanceOf[TrackStateRDDPartition]
+ val prevStateRDDIterator = prevStateRDD.iterator(
+ stateRDDPartition.previousSessionRDDPartition, context)
+ val dataIterator = partitionedDataRDD.iterator(
+ stateRDDPartition.partitionedDataRDDPartition, context)
+
+ val newStateMap = if (prevStateRDDIterator.hasNext) {
+ prevStateRDDIterator.next().stateMap.copy()
+ } else {
+ new EmptyStateMap[K, S]()
+ }
+
+ val emittedRecords = new ArrayBuffer[T]
+
+ val wrappedState = new StateImpl[S]()
+
+ dataIterator.foreach { case (key, value) =>
+ wrappedState.wrap(newStateMap.get(key))
+ val emittedRecord = trackingFunction(key, Some(value), wrappedState)
+ if (wrappedState.isRemoved) {
+ newStateMap.remove(key)
+ } else if (wrappedState.isUpdated) {
+ newStateMap.put(key, wrappedState.get(), currentTime)
+ }
+ emittedRecords ++= emittedRecord
+ }
+
+ if (doFullScan) {
+ if (timeoutThresholdTime.isDefined) {
--- End diff --
done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153311720
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155628691
**[Test build #45572 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45572/consoleFull)** for PR 9256 at commit [`ae64786`](https://github.com/apache/spark/commit/ae64786fd937002a2cc1f80518d54e970a6bbb21).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `sealed abstract class State[S] `\n * `sealed abstract class StateSpec[KeyType, ValueType, StateType, EmittedType] extends Serializable `\n * `case class StateSpecImpl[K, V, S, T](`\n * `sealed abstract class TrackStateDStream[KeyType, ValueType, StateType, EmittedType: ClassTag](`\n * `class InternalTrackStateDStream[K: ClassTag, V: ClassTag, S: ClassTag, E: ClassTag](`\n * ` case class StateInfo[S](`\n * ` class LimitMarker(val num: Int) extends Serializable`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44199760
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ *
+ * @param trackingFunction
+ * @tparam KeyType Class of keys
+ * @tparam ValueType
+ * @tparam StateType
+ * @tparam EmittedType
+ * @return
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ new StateSpecImpl[KeyType, ValueType, StateType, EmittedType](trackingFunction)
+ }
+
+ def create[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ apply(trackingFunction)
+ }
+}
+
+
+/** Internal implementation of [[org.apache.spark.streaming.StateSpec]] interface. */
+private[streaming]
+case class StateSpecImpl[K, V, S: ClassTag, T: ClassTag](
--- End diff --
Oh crap, yes, I should remove them here too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44229996
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
--- End diff --
Why not just
StateSpec.function(..) ?
The extra "new" seems redundant.
It is also more consistent with rest of the SQL code (e.g. window function spec)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44233533
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.language.implicitConversions
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Scala example of using `State`:
+ * {{{
+ * def trackStateFunc(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ *
+ * // Check if state exists
+ * if (state.exists) {
+ *
+ * val existingState = wrappedState.get // Get the existing state
+ *
+ * val shouldRemove = ... // Decide whether to remove the state
+ *
+ * if (shouldRemove) {
+ *
+ * wrappedState.remove() // Remove the state
+ *
+ * } else {
+ *
+ * val newState = ...
+ * wrappedState(newState) // Set the new state
+ *
+ * }
+ * } else {
+ *
+ * val initialState = ...
+ * state.update(initialState) // Set the initial state
+ *
+ * }
+ * }
+ *
+ * }}}
+ *
+ * Java example:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
+ */
+@Experimental
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise it will throw `java.util.NoSuchElementException`.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ *
+ * @throws java.util.NoSuchElementException If the state does not exist.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ *
+ * @throws java.lang.IllegalArgumentException If the state has already been removed, or is
+ * going to be removed
+ */
+ def update(newState: S): Unit
+
+ /**
+ * Remove the state if it exists.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ */
+ def remove(): Unit
+
+ /**
+ * Whether the state is timing out and going to be removed by the system after the current batch.
+ * This timeou can occur if timeout duration has been specified in the
+ * [[org.apache.spark.streaming.StateSpec StatSpec]] and the key has not received any new data
+ * for that timeout duration.
+ */
+ def isTimingOut(): Boolean
+
+ /**
+ * Get the state as an [[scala.Option]]. It will be `Some(state)` if it exists, otherwise `None`.
+ */
+ @inline final def getOption(): Option[S] = if (exists) Some(get()) else None
+
+ @inline final override def toString(): String = {
+ getOption.map { _.toString }.getOrElse("<state not set>")
+ }
+}
+
+private[streaming]
+object State {
+ implicit def toOption[S](state: State[S]): Option[S] = state.getOption()
+}
+
+/** Internal implementation of the [[State]] interface */
+private[streaming] class StateImpl[S] extends State[S] {
--- End diff --
I did it so that we can separate the publicly visible interfaces, from the private[streaming] ones. private[streaming] ones are still publicly accessible from Java, so I was trying to avoid that. Other than than I dont really have a strong reason. If that isnt a strong enough reason to have a separate abstract class and concrete implementation, then I can merge them into one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44197742
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/StateSpec.scala ---
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{HashPartitioner, Partitioner}
+
+
+/**
+ * :: Experimental ::
+ * Abstract class representing all the specifications of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ * Use the [[org.apache.spark.streaming.StateSpec StateSpec.apply()]] or
+ * [[org.apache.spark.streaming.StateSpec StateSpec.create()]] to create instances of
+ * this class.
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+sealed abstract class StateSpec[K, V, S, T] extends Serializable {
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(rdd: RDD[(K, S)]): this.type
+
+ /** Set the RDD containing the initial states that will be used by `trackStateByKey`*/
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ /**
+ * Set the number of partitions by which the state RDDs generated by `trackStateByKey`
+ * will be partitioned. Hash partitioning will be used on the
+ */
+ def numPartitions(numPartitions: Int): this.type
+
+ /**
+ * Set the partitioner by which the state RDDs generated by `trackStateByKey` will be
+ * be partitioned.
+ */
+ def partitioner(partitioner: Partitioner): this.type
+
+ /**
+ * Set the duration after which the state of an idle key will be removed. A key and its state is
+ * considered idle if it has not received any data for at least the given duration. The state
+ * tracking function will be called one final time on the idle states that are going to be
+ * removed; [[org.apache.spark.streaming.State State.isTimingOut()]] set
+ * to `true` in that call.
+ */
+ def timeout(idleDuration: Duration): this.type
+}
+
+
+/**
+ * :: Experimental ::
+ * Builder object for creating instances of [[org.apache.spark.streaming.StateSpec StateSpec]]
+ * that is used for specifying the parameters of the DStream transformation
+ * `trackStateByKey` operation of a
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Example in Scala:
+ * {{{
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream = keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ * }}}
+ *
+ * Example in Java:
+ * {{{
+ * StateStateSpec[StateType, EmittedDataType] spec =
+ * StateStateSpec.create[StateType, EmittedDataType](trackingFunction).numPartition(10);
+ *
+ * JavaDStream[EmittedDataType] emittedRecordDStream =
+ * javaPairDStream.trackStateByKey[StateType, EmittedDataType](spec);
+ * }}}
+ */
+@Experimental
+object StateSpec {
+ /**
+ *
+ * @param trackingFunction
+ * @tparam KeyType Class of keys
+ * @tparam ValueType
+ * @tparam StateType
+ * @tparam EmittedType
+ * @return
+ */
+ def apply[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
+ new StateSpecImpl[KeyType, ValueType, StateType, EmittedType](trackingFunction)
+ }
+
+ def create[KeyType, ValueType, StateType, EmittedType](
+ trackingFunction: (KeyType, Option[ValueType], State[StateType]) => Option[EmittedType]
+ ): StateSpec[KeyType, ValueType, StateType, EmittedType] = {
--- End diff --
Dont need. In fact not having classtag makes this API easy to use from Java. One can easily write
`StateSpec.create<K, V, S, T>(...)` from Java without the javac complaining about additional parameter not being found.
@zsxwing is writing the Java API and he found this as trouble some when using in the Java API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44213899
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.language.implicitConversions
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Scala example of using `State`:
+ * {{{
+ * def trackStateFunc(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ *
+ * // Check if state exists
+ * if (state.exists) {
+ *
+ * val existingState = wrappedState.get // Get the existing state
+ *
+ * val shouldRemove = ... // Decide whether to remove the state
+ *
+ * if (shouldRemove) {
+ *
+ * wrappedState.remove() // Remove the state
+ *
+ * } else {
+ *
+ * val newState = ...
+ * wrappedState(newState) // Set the new state
+ *
+ * }
+ * } else {
+ *
+ * val initialState = ...
+ * state.update(initialState) // Set the initial state
+ *
+ * }
+ * }
+ *
+ * }}}
+ *
+ * Java example:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
+ */
+@Experimental
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise it will throw `java.util.NoSuchElementException`.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ *
+ * @throws java.util.NoSuchElementException If the state does not exist.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ *
+ * @throws java.lang.IllegalArgumentException If the state has already been removed, or is
+ * going to be removed
+ */
+ def update(newState: S): Unit
+
+ /**
+ * Remove the state if it exists.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ */
+ def remove(): Unit
+
+ /**
+ * Whether the state is timing out and going to be removed by the system after the current batch.
+ * This timeou can occur if timeout duration has been specified in the
--- End diff --
Typo: time
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43980906
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/EmittedRecordsDStream.scala ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.dstream
+
+import scala.reflect.ClassTag
+
+import org.apache.spark._
+import org.apache.spark.rdd.{EmptyRDD, RDD}
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.streaming._
+import org.apache.spark.streaming.rdd.{TrackStateRDD, TrackStateRDDRecord}
+
+
+abstract class EmittedRecordsDStream[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
--- End diff --
Very valid concern, that what I needed feedback on. But "EmittedState" implies state is emitted.... which is wrong. How about simply `EmittedDStream`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44067879
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala ---
@@ -350,6 +349,18 @@ class PairDStreamFunctions[K, V](self: DStream[(K, V)])
)
}
+ /** TODO: Add scala docs */
+ def trackStateByKey[S: ClassTag, T: ClassTag](
--- End diff --
For the method signature, since `spec` contains `ClassTag`s for `S` and `T`, we can get it from `spec` and don't need to add `ClassTag` in the method signature.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44238137
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.language.implicitConversions
+
+import org.apache.spark.annotation.Experimental
+
+/**
+ * :: Experimental ::
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * a [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] (Scala) or a
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream JavaPairDStream]] (Java).
+ *
+ * Scala example of using `State`:
+ * {{{
+ * def trackStateFunc(key: String, data: Option[Int], wrappedState: State[Int]): Option[Int] = {
+ *
+ * // Check if state exists
+ * if (state.exists) {
+ *
+ * val existingState = wrappedState.get // Get the existing state
+ *
+ * val shouldRemove = ... // Decide whether to remove the state
+ *
+ * if (shouldRemove) {
+ *
+ * wrappedState.remove() // Remove the state
+ *
+ * } else {
+ *
+ * val newState = ...
+ * wrappedState(newState) // Set the new state
+ *
+ * }
+ * } else {
+ *
+ * val initialState = ...
+ * state.update(initialState) // Set the initial state
+ *
+ * }
+ * }
+ *
+ * }}}
+ *
+ * Java example:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
+ */
+@Experimental
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise it will throw `java.util.NoSuchElementException`.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ *
+ * @throws java.util.NoSuchElementException If the state does not exist.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ *
+ * @throws java.lang.IllegalArgumentException If the state has already been removed, or is
+ * going to be removed
+ */
+ def update(newState: S): Unit
+
+ /**
+ * Remove the state if it exists.
+ *
+ * State cannot be updated if it has been already removed (that is, `remove()` has already been
+ * called) or it is going to be removed due to timeout (that is, `isTimingOut()` is `true`).
+ */
+ def remove(): Unit
+
+ /**
+ * Whether the state is timing out and going to be removed by the system after the current batch.
+ * This timeou can occur if timeout duration has been specified in the
+ * [[org.apache.spark.streaming.StateSpec StatSpec]] and the key has not received any new data
+ * for that timeout duration.
+ */
+ def isTimingOut(): Boolean
+
+ /**
+ * Get the state as an [[scala.Option]]. It will be `Some(state)` if it exists, otherwise `None`.
+ */
+ @inline final def getOption(): Option[S] = if (exists) Some(get()) else None
+
+ @inline final override def toString(): String = {
+ getOption.map { _.toString }.getOrElse("<state not set>")
+ }
+}
+
+private[streaming]
+object State {
+ implicit def toOption[S](state: State[S]): Option[S] = state.getOption()
+}
+
+/** Internal implementation of the [[State]] interface */
+private[streaming] class StateImpl[S] extends State[S] {
+
+ private var state: S = null.asInstanceOf[S]
+ private var defined: Boolean = true
+ private var timingOut: Boolean = false
+ private var updated: Boolean = false
+ private var removed: Boolean = false
+
+ // ========= Public API =========
+ override def exists(): Boolean = {
+ defined
+ }
+
+ override def get(): S = {
+ state
+ }
+
+ override def update(newState: S): Unit = {
+ require(!removed, "Cannot update the state after it has been removed")
+ require(!timingOut, "Cannot update the state that is timing out")
+ state = newState
+ updated = true
+ }
+
+ override def isTimingOut(): Boolean = {
+ timingOut
+ }
+
+ override def remove(): Unit = {
+ require(!timingOut, "Cannot remove the state that is timing out")
+ require(!removed, "Cannot remove the state that has already been removed")
+ removed = true
+ }
+
+ // ========= Internal API =========
+
+ /** Whether the state has been marked for removing */
+ def isRemoved(): Boolean = {
+ removed
+ }
+
+ /** Whether the state has been been updated */
+ def isUpdated(): Boolean = {
+ updated
+ }
+
+ /**
+ * Internal method to update the state data and reset internal flags in `this`.
+ * This method allows `this` object to be reused across many state records.
+ */
+ def wrap(optionalState: Option[S]): Unit = {
+ optionalState match {
+ case Some(newState) =>
+ this.state = newState
+ defined = true
+
+ case None =>
+ this.state = null.asInstanceOf[S]
+ defined = false
+ }
+ timingOut = false
+ removed = false
+ updated = false
+ }
+
+ /**
+ * Internal method to update the state data and reset internal flags in `this`.
+ * This method allows `this` object to be reused across many state records.
+ */
+ def wrapTiminoutState(newState: S): Unit = {
--- End diff --
Good point, will update comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43089007
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/TrackStateSpec.scala ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{HashPartitioner, Partitioner}
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * Abstract class having all the specifications of DStream.trackStateByKey().
+ * Use the `TrackStateSpec.create()` or `TrackStateSpec.create()` to create instances of this class.
+ *
+ * {{{
+ * TrackStateSpec(trackingFunction) // in Scala
+ * TrackStateSpec.create(trackingFunction) // in Java
+ * }}}
+ */
+sealed abstract class TrackStateSpec[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag]
+ extends Serializable {
+
+ def initialState(rdd: RDD[(K, S)]): this.type
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ def numPartitions(numPartitions: Int): this.type
+ def partitioner(partitioner: Partitioner): this.type
+
+ def timeout(interval: Duration): this.type
+}
+
+
+/** Builder object for creating instances of TrackStateSpec */
+object TrackStateSpec {
+
+ def apply[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackingFunction: (K, Option[V], State[S]) => Option[T]): TrackStateSpec[K, V, S, T] = {
+ new TrackStateSpecImpl[K, V, S, T](trackingFunction)
+ }
+
+ def create[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackingFunction: (K, Option[V], State[S]) => Option[T]): TrackStateSpec[K, V, S, T] = {
+ apply(trackingFunction)
+ }
--- End diff --
I think here Java friendly constructor is necessary, `create` might not be directly used in Java code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43095228
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala ---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+/**
+ * Abstract class for getting and updating the tracked state in the `trackStateByKey` operation of
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair DStream]] and
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream]].
+ * {{{
+ *
+ * }}}
+ */
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise wise it will throw an exception.
+ * Check with `exists()` whether the state exists or not before calling `get()`.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value. Note that you cannot update the state if the state is
+ * timing out (that is, `isTimingOut() return true`, or if the state has already been removed by
+ * `remove()`.
+ */
+ def update(newState: S): Unit
+
+ /** Remove the state if it exists. */
+ def remove(): Unit
+
+ /** Is the state going to be timed out by the system after this batch interval */
+ def isTimingOut(): Boolean
+
+ @inline final def getOption(): Option[S] = Option(get())
+
+ /** Get the state if it exists, otherwise return the default value */
+ @inline final def getOrElse[S1 >: S](default: => S1): S1 = {
+ if (exists) this.get else default
+ }
+
+ @inline final override def toString() = getOption.map { _.toString }.getOrElse("<state not set>")
+}
+
+/** Internal implementation of the [[State]] interface */
+private[streaming] class StateImpl[S] extends State[S] {
+
+ private var state: S = null.asInstanceOf[S]
+ private var defined: Boolean = true
+ private var timingOut: Boolean = false
+ private var updated: Boolean = false
+ private var removed: Boolean = false
+
+ // ========= Public API =========
+ def exists(): Boolean = {
+ defined
+ }
+
+ def get(): S = {
+ state
+ }
+
+ def update(newState: S): Unit = {
+ require(!removed, "Cannot update the state after it has been removed")
+ require(!timingOut, "Cannot update the state that is timing out")
+ state = newState
--- End diff --
Yeah, if the user accidentally tries to update the state that is going to be removed by timeout anyways, the system should throw an error rather than silently allowing him/her to update without actually being updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-153311760
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by huitseeker <gi...@git.apache.org>.
Github user huitseeker commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44189624
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala ---
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.util
+
+import java.io.{ObjectInputStream, ObjectOutputStream}
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.SparkConf
+import org.apache.spark.streaming.util.OpenHashMapBasedStateMap._
+import org.apache.spark.util.collection.OpenHashMap
+
+/** Internal interface for defining the map that keeps track of sessions. */
+private[streaming] abstract class StateMap[K: ClassTag, S: ClassTag] extends Serializable {
+
+ /** Get the state for a key if it exists */
+ def get(key: K): Option[S]
+
+ /** Get all the keys and states whose updated time is older than the given threshold time */
+ def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)]
+
+ /** Get all the keys and states in this map. */
+ def getAll(): Iterator[(K, S, Long)]
+
+ /** Add or update state */
+ def put(key: K, state: S, updatedTime: Long): Unit
+
+ /** Remove a key */
+ def remove(key: K): Unit
+
+ /**
+ * Shallow copy `this` map to create a new state map.
+ * Updates to the new map should not mutate `this` map.
+ */
+ def copy(): StateMap[K, S]
+
+ def toDebugString(): String = toString()
+}
+
+/** Companion object for [[StateMap]], with utility methods */
+private[streaming] object StateMap {
+ def empty[K: ClassTag, S: ClassTag]: StateMap[K, S] = new EmptyStateMap[K, S]
+
+ def create[K: ClassTag, S: ClassTag](conf: SparkConf): StateMap[K, S] = {
+ val deltaChainThreshold = conf.getInt("spark.streaming.sessionByKey.deltaChainThreshold",
+ DELTA_CHAIN_LENGTH_THRESHOLD)
+ new OpenHashMapBasedStateMap[K, S](64, deltaChainThreshold)
+ }
+}
+
+/** Specific implementation of SessionStore interface representing an empty map */
+private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMap[K, S] {
+ override def put(key: K, session: S, updateTime: Long): Unit = {
+ throw new NotImplementedError("put() should not be called on an EmptyStateMap")
+ }
+ override def get(key: K): Option[S] = None
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = Iterator.empty
+ override def getAll(): Iterator[(K, S, Long)] = Iterator.empty
+ override def copy(): StateMap[K, S] = new EmptyStateMap[K, S]
+ override def remove(key: K): Unit = { }
+ override def toDebugString(): String = ""
+}
+
+
+
+/** Implementation of StateMap based on Spark's OpenHashMap */
+private[streaming] class OpenHashMapBasedStateMap[K: ClassTag, S: ClassTag](
+ @transient @volatile var parentStateMap: StateMap[K, S],
+ initialCapacity: Int = 64,
+ deltaChainThreshold: Int = DELTA_CHAIN_LENGTH_THRESHOLD
+ ) extends StateMap[K, S] { self =>
+
+ def this(initialCapacity: Int, deltaChainThreshold: Int) = this(
+ new EmptyStateMap[K, S],
+ initialCapacity = initialCapacity,
+ deltaChainThreshold = deltaChainThreshold)
+
+ def this(deltaChainThreshold: Int) = this(
+ initialCapacity = 64, deltaChainThreshold = deltaChainThreshold)
+
+ def this() = this(DELTA_CHAIN_LENGTH_THRESHOLD)
+
+ @transient @volatile private var deltaMap =
+ new OpenHashMap[K, StateInfo[S]](initialCapacity)
+
+ /** Get the session data if it exists */
+ override def get(key: K): Option[S] = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ if (!stateInfo.deleted) {
+ Some(stateInfo.data)
+ } else {
+ None
+ }
+ } else {
+ parentStateMap.get(key)
+ }
+ }
+
+ /** Get all the keys and states whose updated time is older than the give threshold time */
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = {
+ val oldStates = parentStateMap.getByTime(threshUpdatedTime).filter { case (key, value, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.flatMap { case (key, stateInfo) =>
--- End diff --
It makes it harder to not exhaust the iterator (IIUC what you suggest).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44205619
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala ---
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming.util
+
+import java.io.{ObjectInputStream, ObjectOutputStream}
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.SparkConf
+import org.apache.spark.streaming.util.OpenHashMapBasedStateMap._
+import org.apache.spark.util.collection.OpenHashMap
+
+/** Internal interface for defining the map that keeps track of sessions. */
+private[streaming] abstract class StateMap[K: ClassTag, S: ClassTag] extends Serializable {
+
+ /** Get the state for a key if it exists */
+ def get(key: K): Option[S]
+
+ /** Get all the keys and states whose updated time is older than the given threshold time */
+ def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)]
+
+ /** Get all the keys and states in this map. */
+ def getAll(): Iterator[(K, S, Long)]
+
+ /** Add or update state */
+ def put(key: K, state: S, updatedTime: Long): Unit
+
+ /** Remove a key */
+ def remove(key: K): Unit
+
+ /**
+ * Shallow copy `this` map to create a new state map.
+ * Updates to the new map should not mutate `this` map.
+ */
+ def copy(): StateMap[K, S]
+
+ def toDebugString(): String = toString()
+}
+
+/** Companion object for [[StateMap]], with utility methods */
+private[streaming] object StateMap {
+ def empty[K: ClassTag, S: ClassTag]: StateMap[K, S] = new EmptyStateMap[K, S]
+
+ def create[K: ClassTag, S: ClassTag](conf: SparkConf): StateMap[K, S] = {
+ val deltaChainThreshold = conf.getInt("spark.streaming.sessionByKey.deltaChainThreshold",
+ DELTA_CHAIN_LENGTH_THRESHOLD)
+ new OpenHashMapBasedStateMap[K, S](64, deltaChainThreshold)
+ }
+}
+
+/** Specific implementation of SessionStore interface representing an empty map */
+private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMap[K, S] {
+ override def put(key: K, session: S, updateTime: Long): Unit = {
+ throw new NotImplementedError("put() should not be called on an EmptyStateMap")
+ }
+ override def get(key: K): Option[S] = None
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = Iterator.empty
+ override def getAll(): Iterator[(K, S, Long)] = Iterator.empty
+ override def copy(): StateMap[K, S] = new EmptyStateMap[K, S]
+ override def remove(key: K): Unit = { }
+ override def toDebugString(): String = ""
+}
+
+
+
+/** Implementation of StateMap based on Spark's OpenHashMap */
+private[streaming] class OpenHashMapBasedStateMap[K: ClassTag, S: ClassTag](
+ @transient @volatile var parentStateMap: StateMap[K, S],
+ initialCapacity: Int = 64,
+ deltaChainThreshold: Int = DELTA_CHAIN_LENGTH_THRESHOLD
+ ) extends StateMap[K, S] { self =>
+
+ def this(initialCapacity: Int, deltaChainThreshold: Int) = this(
+ new EmptyStateMap[K, S],
+ initialCapacity = initialCapacity,
+ deltaChainThreshold = deltaChainThreshold)
+
+ def this(deltaChainThreshold: Int) = this(
+ initialCapacity = 64, deltaChainThreshold = deltaChainThreshold)
+
+ def this() = this(DELTA_CHAIN_LENGTH_THRESHOLD)
+
+ @transient @volatile private var deltaMap =
+ new OpenHashMap[K, StateInfo[S]](initialCapacity)
+
+ /** Get the session data if it exists */
+ override def get(key: K): Option[S] = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ if (!stateInfo.deleted) {
+ Some(stateInfo.data)
+ } else {
+ None
+ }
+ } else {
+ parentStateMap.get(key)
+ }
+ }
+
+ /** Get all the keys and states whose updated time is older than the give threshold time */
+ override def getByTime(threshUpdatedTime: Long): Iterator[(K, S, Long)] = {
+ val oldStates = parentStateMap.getByTime(threshUpdatedTime).filter { case (key, value, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.flatMap { case (key, stateInfo) =>
+ if (! stateInfo.deleted && stateInfo.updateTime < threshUpdatedTime) {
+ Some((key, stateInfo.data, stateInfo.updateTime))
+ } else None
+ }
+ oldStates ++ updatedStates
+ }
+
+ /** Get all the keys and states in this map. */
+ override def getAll(): Iterator[(K, S, Long)] = {
+
+ val oldStates = parentStateMap.getAll().filter { case (key, _, _) =>
+ !deltaMap.contains(key)
+ }
+
+ val updatedStates = deltaMap.iterator.filter { ! _._2.deleted }.map { case (key, stateInfo) =>
+ (key, stateInfo.data, stateInfo.updateTime)
+ }
+ oldStates ++ updatedStates
+ }
+
+ /** Add or update state */
+ override def put(key: K, state: S, updateTime: Long): Unit = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ stateInfo.update(state, updateTime)
+ } else {
+ deltaMap.update(key, new StateInfo(state, updateTime))
+ }
+ }
+
+ /** Remove a state */
+ override def remove(key: K): Unit = {
+ val stateInfo = deltaMap(key)
+ if (stateInfo != null) {
+ stateInfo.markDeleted()
+ } else {
+ val newInfo = new StateInfo[S](deleted = true)
+ deltaMap.update(key, newInfo)
+ }
+ }
+
+ /**
+ * Shallow copy the map to create a new session store. Updates to the new map
+ * should not mutate `this` map.
+ */
+ override def copy(): StateMap[K, S] = {
+ new OpenHashMapBasedStateMap[K, S](this, deltaChainThreshold = deltaChainThreshold)
+ }
+
+ def shouldCompact: Boolean = {
+ deltaChainLength >= deltaChainThreshold
+ }
+
+ def deltaChainLength: Int = parentStateMap match {
+ case map: OpenHashMapBasedStateMap[_, _] => map.deltaChainLength + 1
+ case _ => 0
+ }
+
+ def approxSize: Int = deltaMap.size + {
+ parentStateMap match {
+ case s: OpenHashMapBasedStateMap[_, _] => s.approxSize
+ case _ => 0
+ }
+ }
+
+ override def toDebugString(): String = {
+ val tabs = if (deltaChainLength > 0) {
+ (" " * (deltaChainLength - 1)) + "+--- "
+ } else ""
+ parentStateMap.toDebugString() + "\n" + deltaMap.iterator.mkString(tabs, "\n" + tabs, "")
+ }
+
+ override def toString(): String = {
+ s"[${System.identityHashCode(this)}, ${System.identityHashCode(parentStateMap)}]"
+ }
+
+ private def writeObject(outputStream: ObjectOutputStream): Unit = {
+
+ outputStream.defaultWriteObject()
+
+ // Write the deltaMap
+ outputStream.writeInt(deltaMap.size)
+ val deltaMapIterator = deltaMap.iterator
+ var deltaMapCount = 0
+ while (deltaMapIterator.hasNext) {
+ deltaMapCount += 1
+ val (key, stateInfo) = deltaMapIterator.next()
+ outputStream.writeObject(key)
+ outputStream.writeObject(stateInfo)
+ }
+ assert(deltaMapCount == deltaMap.size)
+
+ // Write the parentStateMap while consolidating
+ val doCompaction = shouldCompact
+ val newParentSessionStore = if (doCompaction) {
+ val initCapacity = if (approxSize > 0) approxSize else 64
+ new OpenHashMapBasedStateMap[K, S](initialCapacity = initCapacity, deltaChainThreshold)
+ } else { null }
+
+ val iterOfActiveSessions = parentStateMap.getAll()
+
+ var parentSessionCount = 0
+
+ outputStream.writeInt(approxSize)
+
+ while(iterOfActiveSessions.hasNext) {
+ parentSessionCount += 1
+
+ val (key, state, updateTime) = iterOfActiveSessions.next()
+ outputStream.writeObject(key)
+ outputStream.writeObject(state)
+ outputStream.writeLong(updateTime)
+
+ if (doCompaction) {
+ newParentSessionStore.deltaMap.update(
+ key, StateInfo(state, updateTime, deleted = false))
+ }
+ }
+ val limiterObj = new LimitMarker(parentSessionCount)
+ outputStream.writeObject(limiterObj)
+ if (doCompaction) {
+ parentStateMap = newParentSessionStore
+ }
+ }
+
+ private def readObject(inputStream: ObjectInputStream): Unit = {
--- End diff --
note to self: document this method, on what this does.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44045076
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/TrackStateSpec.scala ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{HashPartitioner, Partitioner}
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * Abstract class having all the specifications of DStream.trackStateByKey().
+ * Use the `TrackStateSpec.create()` or `TrackStateSpec.create()` to create instances of this class.
+ *
+ * {{{
+ * TrackStateSpec(trackingFunction) // in Scala
+ * TrackStateSpec.create(trackingFunction) // in Java
+ * }}}
+ */
+sealed abstract class TrackStateSpec[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag]
+ extends Serializable {
+
+ def initialState(rdd: RDD[(K, S)]): this.type
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ def numPartitions(numPartitions: Int): this.type
+ def partitioner(partitioner: Partitioner): this.type
+
+ def timeout(interval: Duration): this.type
--- End diff --
Is this scala Duration? Why not just take milliseconds?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9256#issuecomment-155440818
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org