[jira] [Created] (FLINK-37025) Periodic SQL watermarks can travel back in time

["Dawid Wysakowicz (Jira)" <ji...@apache.org>]

Dawid Wysakowicz created FLINK-37025:
----------------------------------------

             Summary: Periodic SQL watermarks can travel back in time
                 Key: FLINK-37025
                 URL: https://issues.apache.org/jira/browse/FLINK-37025
             Project: Flink
          Issue Type: Bug
          Components: Table SQL / Runtime
    Affects Versions: 2.0-preview, 1.20.0
            Reporter: Dawid Wysakowicz
             Fix For: 2.0.0, 1.20.1


If watermarks are generated through SQL, e.g. using `WATERMARK FOR ts AS ts` is used https://github.com/apache/flink/blob/e9353319ad625baa5b2c20fa709ab5b23f83c0f4/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/generated/GeneratedWatermarkGeneratorSupplier.java#L86

The code in {{onEvent}} can move the {{currentWatermark}} back in time, because it does not check if it advanced. This is fine in case of `on-event` watermarks`, because underlying  {{WatermarkOutput}} will, but it has confusing outcomes for `on-periodic`.

Example:
If events with timestamps:
```
{"id":1,"ts":"2024-01-01 00:00:00"}
{"id":3,"ts":"2024-01-03 00:00:00"}
{"id":2,"ts":"2024-01-02 00:00:00"}
```
come within the periodic emit interval, the generated watermark will be "2024-01-02 00:00:00" instead of "2024-01-03 00:00:00". As a result the watermark for "2024-01-03 00:00:00" is swallowed and a new event is required to progress the processing of that event.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)