Window over events defined by a time range

[Padarn Wilson <pa...@grab.com>]

Hi all,

I'm trying to figure out what the "Flink" way of achieving what I'd like to
is.

Imagine I have the following three events in my stream

event1: {"start_time": 0, "end_time": 1, "value": "a"}
event2: {"start_time": 0, "end_time": 2, "value": "b"}
event3: {"start_time": 1, "end_time": 2, "value": "c"}

From this I would like to create tumbling windows of length 1, that collect
all the events which overlap that window and make a list from the "values"
seen, so the result would be two windows

window1 [0, 1] = {"a", "b"}
window2 [1, 2] ={"b", "c"}

However my understanding is that because my original stream only has three
events, no matter how I create windows, I cannot have event 2 in both
windows.

I can think of how this could be done by mapping each event into a separate
event for the windows that it should fall into, i.e we split event 2 into

event2a: {"start_time": 0, "end_time": 1, "value": "b"}
event2b: {"start_time": 1, "end_time": 2, "value": "b"}

But this seems awfully cumbersome when the logic is more complicated.

Is there a natural way to do this window overlapping windows?

-- 
_Grab is hiring. Learn more at *https://grab.careers 
<https://grab.careers/>*_


By communicating with Grab Inc and/or its 
subsidiaries, associate companies and jointly controlled entities (“Grab 
Group”), you are deemed to have consented to processing of your personal 
data as set out in the Privacy Notice which can be viewed at 
https://grab.com/privacy/ <https://grab.com/privacy/>


This email contains 
confidential information and is only for the intended recipient(s). If you 
are not the intended recipient(s), please do not disseminate, distribute or 
copy this email and notify Grab Group immediately if you have received this 
by mistake and delete this email from your system. Email transmission 
cannot be guaranteed to be secure or error-free as any information therein 
could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or 
contain viruses. Grab Group do not accept liability for any errors or 
omissions in the contents of this email arises as a result of email 
transmission. All intellectual property rights in this email and 
attachments therein shall remain vested in Grab Group, unless otherwise 
provided by law.

Re: Window over events defined by a time range

[Fabian Hueske <fh...@gmail.com>]

Hi,

Flink can add events to mulitple windows. For instance, the built-in
sliding windows are doing this.
You can address your use case by implementing a custom WindowAssigner [1].

Best, Fabian

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/windows.html#window-assigners

2018-05-03 8:39 GMT+02:00 Padarn Wilson <pa...@grab.com>:

> Hi all,
>
> I'm trying to figure out what the "Flink" way of achieving what I'd like
> to is.
>
> Imagine I have the following three events in my stream
>
> event1: {"start_time": 0, "end_time": 1, "value": "a"}
> event2: {"start_time": 0, "end_time": 2, "value": "b"}
> event3: {"start_time": 1, "end_time": 2, "value": "c"}
>
> From this I would like to create tumbling windows of length 1, that
> collect all the events which overlap that window and make a list from the
> "values" seen, so the result would be two windows
>
> window1 [0, 1] = {"a", "b"}
> window2 [1, 2] ={"b", "c"}
>
> However my understanding is that because my original stream only has three
> events, no matter how I create windows, I cannot have event 2 in both
> windows.
>
> I can think of how this could be done by mapping each event into a
> separate event for the windows that it should fall into, i.e we split event
> 2 into
>
> event2a: {"start_time": 0, "end_time": 1, "value": "b"}
> event2b: {"start_time": 1, "end_time": 2, "value": "b"}
>
> But this seems awfully cumbersome when the logic is more complicated.
>
> Is there a natural way to do this window overlapping windows?
>
>
>
> *Grab is hiring. Learn more at **https://grab.careers
> <https://grab.careers/>*
>
> By communicating with Grab Inc and/or its subsidiaries, associate
> companies and jointly controlled entities (“Grab Group”), you are deemed to
> have consented to processing of your personal data as set out in the
> Privacy Notice which can be viewed at https://grab.com/privacy/
>
> This email contains confidential information and is only for the intended
> recipient(s). If you are not the intended recipient(s), please do not
> disseminate, distribute or copy this email and notify Grab Group
> immediately if you have received this by mistake and delete this email from
> your system. Email transmission cannot be guaranteed to be secure or
> error-free as any information therein could be intercepted, corrupted,
> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do
> not accept liability for any errors or omissions in the contents of this
> email arises as a result of email transmission. All intellectual property
> rights in this email and attachments therein shall remain vested in Grab
> Group, unless otherwise provided by law.
>