You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Something Something <ma...@gmail.com> on 2020/02/27 23:17:18 UTC

Spark Streaming: Aggregating values across batches

We've a Spark Streaming job that calculates some values in each batch. What
we need to do now is aggregate values across ALL batches. What is the best
strategy to do this in Spark Streaming. Should we use 'Spark Accumulators'
for this?

Re: Spark Streaming: Aggregating values across batches

Posted by Tathagata Das <ta...@gmail.com>.

Use Structured Streaming. Its aggregation, by definition, is across batches.

On Thu, Feb 27, 2020 at 3:17 PM Something Something <
mailinglists19@gmail.com> wrote:

> We've a Spark Streaming job that calculates some values in each batch.
> What we need to do now is aggregate values across ALL batches. What is the
> best strategy to do this in Spark Streaming. Should we use 'Spark
> Accumulators' for this?
>