You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by micvog <mi...@micvog.com> on 2015/07/10 13:30:21 UTC

Best way to avoid updateStateByKey from running without data

UpdateStateByKey will run the update function on every interval, even if the
incoming batch is empty. Is there a way to prevent that? If the incoming
DStream contains no RDDs (or RDDs of count 0) then I don't want my update
function to run.

Note that this is different from running the update function and writing an
if (!newValues.isEmpty) inside the function.

Thanks,
Michael



-----
Michael Vogiatzis
@mvogiatzis 
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Best-way-to-avoid-updateStateByKey-from-running-without-data-tp23762.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org