How Flink distinguishes between late and in-time events?

[Ori Popowski <or...@gmail.com>]

In the documentation
<https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#allowed-lateness>
it states that:

*[…], Flink keeps the state of windows until their allowed lateness
expires. Once this happens, Flink removes the window and deletes its state,
as also described in the Window Lifecycle
<https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#window-lifecycle>
section.*

However, something doesn't make sense to me.

If Flink deletes the window state, then how can it know that subsequent
events are late? i.e. if the state is deleted, then Flink has no way of
knowing than an event is late, because it can think it's just a new event,
unless it keeps track of which keyed windows are closed forever.

Does Flink remember which keyed windows are closed forever?

Thanks.

Re: How Flink distinguishes between late and in-time events?

[Ori Popowski <or...@gmail.com>]

That makes sense. Thanks

On Thu, Aug 20, 2020 at 7:45 PM Piotr Nowojski <pn...@apache.org> wrote:

> Hi Ori,
>
> No. Flink does it differently. Operators that are keeping track of late
> events, are remembering the latest watermark. If a new element arrives with
> even time lower compared to the latest watermark, it is marked as a late
> event [1]
>
> Piotrek
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.11/concepts/timely-stream-processing.html#lateness
>
> czw., 20 sie 2020 o 17:13 Ori Popowski <or...@gmail.com> napisał(a):
>
>> In the documentation
>> <https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#allowed-lateness>
>> it states that:
>>
>> *[…], Flink keeps the state of windows until their allowed lateness
>> expires. Once this happens, Flink removes the window and deletes its state,
>> as also described in the Window Lifecycle
>> <https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#window-lifecycle>
>> section.*
>>
>> However, something doesn't make sense to me.
>>
>> If Flink deletes the window state, then how can it know that subsequent
>> events are late? i.e. if the state is deleted, then Flink has no way of
>> knowing than an event is late, because it can think it's just a new event,
>> unless it keeps track of which keyed windows are closed forever.
>>
>> Does Flink remember which keyed windows are closed forever?
>>
>> Thanks.
>>
>

Re: How Flink distinguishes between late and in-time events?

[Piotr Nowojski <pn...@apache.org>]

Hi Ori,

No. Flink does it differently. Operators that are keeping track of late
events, are remembering the latest watermark. If a new element arrives with
even time lower compared to the latest watermark, it is marked as a late
event [1]

Piotrek

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.11/concepts/timely-stream-processing.html#lateness

czw., 20 sie 2020 o 17:13 Ori Popowski <or...@gmail.com> napisał(a):

> In the documentation
> <https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#allowed-lateness>
> it states that:
>
> *[…], Flink keeps the state of windows until their allowed lateness
> expires. Once this happens, Flink removes the window and deletes its state,
> as also described in the Window Lifecycle
> <https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#window-lifecycle>
> section.*
>
> However, something doesn't make sense to me.
>
> If Flink deletes the window state, then how can it know that subsequent
> events are late? i.e. if the state is deleted, then Flink has no way of
> knowing than an event is late, because it can think it's just a new event,
> unless it keeps track of which keyed windows are closed forever.
>
> Does Flink remember which keyed windows are closed forever?
>
> Thanks.
>