Emulate Tumbling window in Event Time Space

[Dhruv Kumar <ga...@gmail.com>]

Hi

I was trying to emulate tumbling window in event time space. Here is the link to my code.
I am using the process function to do the custom processing which I want to do within every window. I am having an issue of how to emit results at the end of every window since my watermark only gets emitted at every incoming event (incoming event will mostly not intersect with the end time of any window). Seems like I need to add a trigger somewhere which fires at the end of every window. Could any one here help me? Sorry, if I am not clear in anything. I am quite new to Flink. 

Thanks
Dhruv

Re: Emulate Tumbling window in Event Time Space

[Piotr Nowojski <pi...@data-artisans.com>]

Hi,

As Xingcan responded, you could use already built in operator for that. 

If you really want to implement something on your own (need custom feature? For fun?), you would have to implement some variation of a InternalTimerService from Flink (you can browse the code for an inspiration). On each processed element you have to keep updating state of your in memory/in state windows with timestamps marking when they should be triggered. Then on each processed watermark in your operator you need to trigger/fire windows matching to the processed watermark.

Piotrek

> On 9 Mar 2018, at 07:50, Xingcan Cui <xi...@gmail.com> wrote:
> 
> Hi Dhruv,
> 
> there’s no need to implement the window logic with the low-level `ProcessFunction` yourself. Flink has provided built-in window operators and you just need to implement the `WindowFunction` for that [1].
> 
> Best,
> Xingcan
> 
> [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#window-functions <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#window-functions>
> 
>> On 9 Mar 2018, at 1:51 PM, Dhruv Kumar <gargdhruv36@gmail.com <ma...@gmail.com>> wrote:
>> 
>> Hi
>> 
>> I was trying to emulate tumbling window in event time space. Here <https://github.com/kudhru/flink-learning/blob/master/TumblingWindowEmulationShare.java> is the link to my code.
>> I am using the process function to do the custom processing which I want to do within every window. I am having an issue of how to emit results at the end of every window since my watermark only gets emitted at every incoming event (incoming event will mostly not intersect with the end time of any window). Seems like I need to add a trigger somewhere which fires at the end of every window. Could any one here help me? Sorry, if I am not clear in anything. I am quite new to Flink. 
>> 
>> Thanks
>> Dhruv
> 

Re: Emulate Tumbling window in Event Time Space

[Xingcan Cui <xi...@gmail.com>]

Hi Dhruv,

there’s no need to implement the window logic with the low-level `ProcessFunction` yourself. Flink has provided built-in window operators and you just need to implement the `WindowFunction` for that [1].

Best,
Xingcan

[1] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#window-functions <https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/windows.html#window-functions>

> On 9 Mar 2018, at 1:51 PM, Dhruv Kumar <ga...@gmail.com> wrote:
> 
> Hi
> 
> I was trying to emulate tumbling window in event time space. Here <https://github.com/kudhru/flink-learning/blob/master/TumblingWindowEmulationShare.java> is the link to my code.
> I am using the process function to do the custom processing which I want to do within every window. I am having an issue of how to emit results at the end of every window since my watermark only gets emitted at every incoming event (incoming event will mostly not intersect with the end time of any window). Seems like I need to add a trigger somewhere which fires at the end of every window. Could any one here help me? Sorry, if I am not clear in anything. I am quite new to Flink. 
> 
> Thanks
> Dhruv