Spark Streaming, external windowing?

[Sargun Dhillon <sa...@sargun.me>]

Does anyone here have a way to do Spark Streaming with external timing
for windows? Right now, it relies on the wall clock of the driver to
determine the amount of time that each batch read lasts.

We have a Kafka, and HDFS ingress into our Spark Streaming pipeline
where the events are annotated by the timestamps that they happened
(in real time) in. We would like to keep our windows based on those
timestamps, as opposed to based on the driver time.

Does anyone have any ideas how to do this?

Re: Spark Streaming, external windowing?

[Tathagata Das <ta...@gmail.com>]

One way to do that is currently possible is given here

http://mail-archives.apache.org/mod_mbox/spark-user/201407.mbox/%3CCAMwrk0=B38DewYSLiWYc6HMZe8tTy8innbw6ixATnD1Ue2-amg@mail.gmail.com%3E


On Wed, Jul 16, 2014 at 1:16 AM, Gerard Maas <ge...@gmail.com> wrote:

> Hi Sargun,
>
> There have been few discussions on the list recently about the topic. The
> short answer is that this is not supported at the moment.
> This is a particularly good thread as it discusses the current state and
> limitations:
>
>
> http://apache-spark-developers-list.1001551.n3.nabble.com/brainsotrming-Generalization-of-DStream-a-ContinuousRDD-td7349.html
>
> -kr, Gerard.
>
>
> On Wed, Jul 16, 2014 at 9:56 AM, Sargun Dhillon <sa...@sargun.me> wrote:
>
>> Does anyone here have a way to do Spark Streaming with external timing
>> for windows? Right now, it relies on the wall clock of the driver to
>> determine the amount of time that each batch read lasts.
>>
>> We have a Kafka, and HDFS ingress into our Spark Streaming pipeline
>> where the events are annotated by the timestamps that they happened
>> (in real time) in. We would like to keep our windows based on those
>> timestamps, as opposed to based on the driver time.
>>
>> Does anyone have any ideas how to do this?
>>
>
>

Re: Spark Streaming, external windowing?

[Gerard Maas <ge...@gmail.com>]

Hi Sargun,

There have been few discussions on the list recently about the topic. The
short answer is that this is not supported at the moment.
This is a particularly good thread as it discusses the current state and
limitations:

http://apache-spark-developers-list.1001551.n3.nabble.com/brainsotrming-Generalization-of-DStream-a-ContinuousRDD-td7349.html

-kr, Gerard.


On Wed, Jul 16, 2014 at 9:56 AM, Sargun Dhillon <sa...@sargun.me> wrote:

> Does anyone here have a way to do Spark Streaming with external timing
> for windows? Right now, it relies on the wall clock of the driver to
> determine the amount of time that each batch read lasts.
>
> We have a Kafka, and HDFS ingress into our Spark Streaming pipeline
> where the events are annotated by the timestamps that they happened
> (in real time) in. We would like to keep our windows based on those
> timestamps, as opposed to based on the driver time.
>
> Does anyone have any ideas how to do this?
>