[jira] [Closed] (FLINK-33077) Minimize the risk of hard back-pressure with buffer debloating enabled

["Roman Khachatryan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-33077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Roman Khachatryan closed FLINK-33077.
-------------------------------------
    Resolution: Fixed

Fixed in c7f6470bb8cc314e7651b03e171af057f4edec1e.

> Minimize the risk of hard back-pressure with buffer debloating enabled
> ----------------------------------------------------------------------
>
>                 Key: FLINK-33077
>                 URL: https://issues.apache.org/jira/browse/FLINK-33077
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Checkpointing
>            Reporter: Stefan Richter
>            Assignee: Stefan Richter
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.18.0
>
>
> {*}Problem{*}:
> Buffer debloating sets buffer size to {{256}} bytes because of back-pressure.
> Such small buffers might not be enough to emit the processing results of a single record. The task thread would request new buffers, and often block.
> That results in significant checkpoint delays (up to minutes instead of seconds).
> Adding more overdraft buffers helps, but depends on the job DoP
> Raising {{taskmanager.memory.min-segment-size}} from {{256}} helps, but depends on the multiplication factor of the operator.
> {*}Solution{*}:
>  * Ignore Buffer Debloater hints and extend the buffer if possible - when this prevents emitting an output record fully AND this is the last available buffer.
>  * Prevent the subsequent flush of the buffer so that more output records can be emitted (flatMap-like and join operators)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)