Change In Behavior for EventTimeSessionWindow between Flink 1.11.1 and 1.14.0

[Jai Patel <ja...@cloudkitchens.com>]

Hey all,

I observed what appears to be a change in behavior for
EventTimeSessionWindows when upgrading from 1.11.1 to 1.14.0. This was
identified in a unit test.

For a window with a defined time gap of 10 seconds.

   - Publish KEY_1 with eventtime 1 second
   - Publish KEY_1 with eventtime 3 seconds
   - Publish KEY_1 with eventtime 2 seconds
   - Publish KEY_2 with eventtime 101 seconds

For Flink 1.11.1 the window for KEY_1 closes, reduces, and publishes,
supposedly because KEY_2 had an event time greater than 10 seconds after
the last message in KEY_1's window. KEY_2 window would also not close.  In
the absence of KEY_2 the KEY_1 window would not close.

For Flink 1.14.0 the main difference is that the window for KEY_2 *DOES*
close even though there are no new messages after 111 seconds.

This appears to be a change in behavior. The nearest I could find was
https://issues.apache.org/jira/browse/FLINK-20443 but that’s in 1.14.1. Is
there an explanation for this change in behavior? Is it expected or
desirable? Is it because all pending windows are automatically closed based
on an updated trigger behavior?

I tested the same behavior for ProcessingTimeSessionWindows and did not
observe a similar change in behavior.

Thanks.
Jai

Re: Change In Behavior for EventTimeSessionWindow between Flink 1.11.1 and 1.14.0

[Caizhi Weng <ts...@gmail.com>]

Hi!

Did you specify source idleness time in watermark strategy? If yes source
will be marked idle if no records are sent for a period of time, causing
the 1.14 behavior you described. Could you also share your unit test case
so we can take a deeper look?

Jai Patel <ja...@cloudkitchens.com> 于2021年11月12日周五 上午3:33写道：

> Hey all,
>
> I observed what appears to be a change in behavior for
> EventTimeSessionWindows when upgrading from 1.11.1 to 1.14.0. This was
> identified in a unit test.
>
> For a window with a defined time gap of 10 seconds.
>
>    - Publish KEY_1 with eventtime 1 second
>    - Publish KEY_1 with eventtime 3 seconds
>    - Publish KEY_1 with eventtime 2 seconds
>    - Publish KEY_2 with eventtime 101 seconds
>
> For Flink 1.11.1 the window for KEY_1 closes, reduces, and publishes,
> supposedly because KEY_2 had an event time greater than 10 seconds after
> the last message in KEY_1's window. KEY_2 window would also not close.  In
> the absence of KEY_2 the KEY_1 window would not close.
>
> For Flink 1.14.0 the main difference is that the window for KEY_2 *DOES*
> close even though there are no new messages after 111 seconds.
>
> This appears to be a change in behavior. The nearest I could find was
> https://issues.apache.org/jira/browse/FLINK-20443 but that’s in 1.14.1.
> Is there an explanation for this change in behavior? Is it expected or
> desirable? Is it because all pending windows are automatically closed based
> on an updated trigger behavior?
>
> I tested the same behavior for ProcessingTimeSessionWindows and did not
> observe a similar change in behavior.
>
> Thanks.
> Jai
>