You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Udo Fholl <ud...@gmail.com> on 2016/02/03 13:01:10 UTC
Spark streaming archive results
Hi,
I want to aggregate on a small window and send downstream every 30 secs.
But I would also like to store in our archive the outcome every 20min.
My current approach (simplified version) is:
val stream = //
val statedStream = stream.mapWithState(stateSpec)
val archiveStream = statedStream
statedStream.foreachRDD(rdd => rdd.foreachPartition(d =>
sendDownstream(d)))
archive.window(Minutes(20).foreachRDD(rdd => rdd.foreachPartition(d =>
archive(d)))
Is there a more suitable way to deal with this situations?
Thanks.
Udo