[jira] [Commented] (BEAM-10691) FlinkRunner: pipeline might get stuck due to timer watermark hold not being released

["Maximilian Michels (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-10691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17177645#comment-17177645 ] 

Maximilian Michels commented on BEAM-10691:
-------------------------------------------

Putting in triage mode again because there is not enough information here yet to indicate this is actually a bug. As discussed in https://github.com/apache/beam/pull/12551, there might be a timer set that is yet to fire in the future because the input watermark has not reached the fire timestamp yet. If this timer has an output timestamp configured, the output watermark will be held back. This is a common source of confusion when it comes to the output timestamp feature.

> FlinkRunner: pipeline might get stuck due to timer watermark hold not being released
> ------------------------------------------------------------------------------------
>
>                 Key: BEAM-10691
>                 URL: https://issues.apache.org/jira/browse/BEAM-10691
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>    Affects Versions: 2.23.0, 2.24.0
>            Reporter: Jan Lukavský
>            Assignee: Jan Lukavský
>            Priority: P1
>          Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> Pipeline might stop progressing watermark in certain cases due to timer output timestamp not being released from FlinkTimerInternals#outputTimestampQueue. The pipeline has to be restarted from checkpoint to reload the cache and free watermark hold.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)