You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Piotr Nowojski (Jira)" <ji...@apache.org> on 2025/02/27 16:06:00 UTC
[jira] [Updated] (FLINK-37399) Watermark alignment can prevent backlogged jobs from using all available resources
[ https://issues.apache.org/jira/browse/FLINK-37399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Piotr Nowojski updated FLINK-37399:
-----------------------------------
Description:
Imagine the following scenario. Max allowed watermark alignment drift is set to 30s and watermark alignment is synced/announced across subtasks every 1s. This makes perfect sense during normal records processing.
But when processing messages from backlog, it creates a problem, defacto capping how quickly watermarks can be progressing. For example:
* at {{t0}} we announce that max allowed watermark is {{max_w0 = min(reported_watermark) + 30s}}
* next announcement will happen at {{t1 = t0 + 1s }}
* any source/partition exceeds {{max_w0}} until the next announcement happens at {{t1}}, it will be blocked by the watermark alignment
In other words, with the above configuration (30s allowed drift announced every ~1s), we are de facto capping the backlog processing speed to:
*30 "event time" seconds per each 1 "real world" second*
The problem is the delay in how quickly watermark announcements are propagating between operators and the coordinator.
was:
Imagine the following scenario. Max allowed watermark alignment drift is set to 30s and watermark alignment is synced/announced across subtasks every 1s. This makes perfect sense during normal records processing.
But when processing messages from backlog, it creates a problem, defacto capping how quickly watermarks can be progressing. For example:
* at {{t0}} we announce that max allowed watermark is {{max_w0 = min(reported_watermark) + 30s}}
* next announcement will happen at {{t1 = t0 + 1s }}
* any source/partition exceeds {{max_w0}} until the next announcement happens at {{t1}}, it will be blocked by the watermark alignment
In other words, with the above configuration (30s allowed drift announced every ~1s), we are de facto capping the backlog processing speed to:
*30 "event time" seconds per each 1 "real world" second*
> Watermark alignment can prevent backlogged jobs from using all available resources
> ----------------------------------------------------------------------------------
>
> Key: FLINK-37399
> URL: https://issues.apache.org/jira/browse/FLINK-37399
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
> Affects Versions: 2.0.0, 1.18.1, 1.19.2
> Reporter: Piotr Nowojski
> Assignee: Piotr Nowojski
> Priority: Major
>
> Imagine the following scenario. Max allowed watermark alignment drift is set to 30s and watermark alignment is synced/announced across subtasks every 1s. This makes perfect sense during normal records processing.
> But when processing messages from backlog, it creates a problem, defacto capping how quickly watermarks can be progressing. For example:
> * at {{t0}} we announce that max allowed watermark is {{max_w0 = min(reported_watermark) + 30s}}
> * next announcement will happen at {{t1 = t0 + 1s }}
> * any source/partition exceeds {{max_w0}} until the next announcement happens at {{t1}}, it will be blocked by the watermark alignment
> In other words, with the above configuration (30s allowed drift announced every ~1s), we are de facto capping the backlog processing speed to:
> *30 "event time" seconds per each 1 "real world" second*
> The problem is the delay in how quickly watermark announcements are propagating between operators and the coordinator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)