[jira] [Assigned] (FLINK-37025) Periodic SQL watermarks can travel back in time

["Dawid Wysakowicz (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-37025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dawid Wysakowicz reassigned FLINK-37025:
----------------------------------------

    Assignee: Dawid Wysakowicz

> Periodic SQL watermarks can travel back in time
> -----------------------------------------------
>
>                 Key: FLINK-37025
>                 URL: https://issues.apache.org/jira/browse/FLINK-37025
>             Project: Flink
>          Issue Type: Bug
>          Components: Table SQL / Runtime
>    Affects Versions: 1.20.0, 2.0-preview
>            Reporter: Dawid Wysakowicz
>            Assignee: Dawid Wysakowicz
>            Priority: Major
>             Fix For: 2.0.0, 1.20.1
>
>
> If watermarks are generated through SQL, e.g. using {{WATERMARK FOR ts AS ts}} is used https://github.com/apache/flink/blob/e9353319ad625baa5b2c20fa709ab5b23f83c0f4/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/generated/GeneratedWatermarkGeneratorSupplier.java#L86
> The code in {{onEvent}} can move the {{currentWatermark}} back in time, because it does not check if it advanced. This is fine in case of {{on-event}} watermarks, because underlying  {{WatermarkOutput}} will, but it has confusing outcomes for {{on-periodic}}.
> Example:
> If events with timestamps:
> {code}
> {"id":1,"ts":"2024-01-01 00:00:00"}
> {"id":3,"ts":"2024-01-03 00:00:00"}
> {"id":2,"ts":"2024-01-02 00:00:00"}
> {code}
> come within the periodic emit interval, the generated watermark will be "2024-01-02 00:00:00" instead of "2024-01-03 00:00:00". As a result the watermark for "2024-01-03 00:00:00" is swallowed and a new event is required to progress the processing of that event.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)