Re: mapWithState: remove key

[Udo Fholl <ud...@gmail.com>]

That makes sense. Thanks for your quick response.

On Fri, Jan 29, 2016 at 7:01 PM, Shixiong(Ryan) Zhu <shixiong@databricks.com
> wrote:

> 1. To remove a state, you need to call "state.remove()". If you return a
> None in the function, it just means don't output it as the DStream's
> output, but the state won't be removed if you don't call "state.remove()".
>
> 2. For NoSuchElementException, here is the doc for "State.get":
>
>   /**
>    * Get the state if it exists, otherwise it will throw
> `java.util.NoSuchElementException`.
>    * Check with `exists()` whether the state exists or not before calling
> `get()`.
>    *
>    * @throws java.util.NoSuchElementException If the state does not exist.
>    */
>
>
>
>
> On Fri, Jan 29, 2016 at 10:45 AM, Udo Fholl <ud...@gmail.com>
> wrote:
>
>> Hi,
>>
>> From the signature of the "mapWithState" method I infer that by returning
>> a "None.type" (in Scala) the key is removed from the state. Is that so?
>> Sorry if it is in the docs, but it wasn't entirely clear to me.
>>
>> I'm chaining operations and calling "mapWithState" twice (one to
>> consolidate, then I perform some operations that might, or might not
>> succeed, and invoke "mapWithState" again). I'm getting this error[1] which
>> I suppose is because I'm returning "None" in the "mapWithState" function.
>>
>> Thank you.
>>
>> Best regards,
>> Udo.
>>
>> [1]: java.util.NoSuchElementException: State is not set
>>     at org.apache.spark.streaming.StateImpl.get(State.scala:150)
>>
>
>