You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Udo Fholl <ud...@gmail.com> on 2016/02/03 13:01:10 UTC

Spark streaming archive results

Hi,

I want to aggregate on a small window and send downstream every 30 secs.
But I would also like to store in our archive the outcome every 20min.

My current approach (simplified version) is:

  val stream = //
  val statedStream = stream.mapWithState(stateSpec)
  val archiveStream = statedStream


  statedStream.foreachRDD(rdd => rdd.foreachPartition(d =>
sendDownstream(d)))

  archive.window(Minutes(20).foreachRDD(rdd => rdd.foreachPartition(d =>
archive(d)))


Is there a more suitable way to deal with this situations?

Thanks.

Udo