You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by micvog <mi...@micvog.com> on 2015/07/03 13:34:10 UTC
Spark Streaming broadcast to all keys
UpdateStateByKey is useful but what if I want to perform an operation to all
existing keys (not only the ones in this RDD).
Word count for example - is there a way to decrease *all* words seen so far
by 1?
I was thinking of keeping a static class per node with the count information
and issuing a broadcast command to take a certain action, but could not find
a broadcast-to-all-nodes functionality or a better way.
Thanks,
Michael
-----
Michael Vogiatzis
@mvogiatzis
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-broadcast-to-all-keys-tp23609.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark Streaming broadcast to all keys
Posted by Silvio Fiorito <si...@granturing.com>.
updateStateByKey will run for all keys, whether they have new data in a batch or not so you should be able to still use it.
On 7/3/15, 7:34 AM, "micvog" <mi...@micvog.com> wrote:
>UpdateStateByKey is useful but what if I want to perform an operation to all
>existing keys (not only the ones in this RDD).
>
>Word count for example - is there a way to decrease *all* words seen so far
>by 1?
>
>I was thinking of keeping a static class per node with the count information
>and issuing a broadcast command to take a certain action, but could not find
>a broadcast-to-all-nodes functionality or a better way.
>
>Thanks,
>Michael
>
>
>
>-----
>Michael Vogiatzis
>@mvogiatzis
>--
>View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-broadcast-to-all-keys-tp23609.html
>Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>For additional commands, e-mail: user-help@spark.apache.org
>