You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "shicheng31604@gmail.com" <sh...@gmail.com> on 2019/03/11 08:42:24 UTC

returning type of function that needs to be passed to method 'mapWithState'

Hi all:
    In the `mapWithState`    method in spark streaming, you need to pass in an anonymous function. This function maintains a state and should return a  result. It can be said that the final stateful result can be obtained from the state object.
        So, what is the significance of returning result? 
        I looked up the official API, and it  did not specifically say that the result is used for,    just give a simple explanation, as follows:

    

    // A mapping function that maintains an integer state and return a String
    def mappingFunction(key: String, value: Option[Int], state: State[Int]): Option[String] = {
      // Use state.exists(), state.get(), state.update() and state.remove()
      // to manage state, and return the necessary string
    }
    val spec = StateSpec.function(mappingFunction).numPartitions(10)
    val mapWithStateDStream = keyValueDStream.mapWithState[StateType, MappedType](spec)
   Can anyone help me with this problem?Thanks!



   


shicheng31604@gmail.com