You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "shicheng31604@gmail.com" <sh...@gmail.com> on 2019/03/11 08:42:24 UTC
returning type of function that needs to be passed to method 'mapWithState'
Hi all:
In the `mapWithState` method in spark streaming, you need to pass in an anonymous function. This function maintains a state and should return a result. It can be said that the final stateful result can be obtained from the state object.
So, what is the significance of returning result?
I looked up the official API, and it did not specifically say that the result is used for, just give a simple explanation, as follows:
// A mapping function that maintains an integer state and return a String
def mappingFunction(key: String, value: Option[Int], state: State[Int]): Option[String] = {
// Use state.exists(), state.get(), state.update() and state.remove()
// to manage state, and return the necessary string
}
val spec = StateSpec.function(mappingFunction).numPartitions(10)
val mapWithStateDStream = keyValueDStream.mapWithState[StateType, MappedType](spec)
Can anyone help me with this problem?Thanks!
shicheng31604@gmail.com