You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Josh J <jo...@gmail.com> on 2014/10/07 21:50:39 UTC

dynamic sliding window duration

Hi,

I have a source which fluctuates in the frequency of streaming tuples. I
would like to process certain batch counts, rather than batch window
durations. Is it possible to either

1) define batch window sizes
or
2) dynamically adjust the duration of the sliding window?

Thanks,
Josh

Re: dynamic sliding window duration

Posted by Tobias Pfeiffer <tg...@preferred.jp>.

Hi,

On Wed, Oct 8, 2014 at 4:50 AM, Josh J <jo...@gmail.com> wrote:
>
> I have a source which fluctuates in the frequency of streaming tuples. I
> would like to process certain batch counts, rather than batch window
> durations. Is it possible to either
>
> 1) define batch window sizes
>

Cf.
http://apache-spark-user-list.1001560.n3.nabble.com/window-every-n-elements-instead-of-time-based-td2085.html

> 2) dynamically adjust the duration of the sliding window?
>

That's not possible AFAIK, because you can't change anything in the
processing pipeline after StreamingContext has been started.

Tobias