[jira] [Closed] (FLINK-2497) Rework streaming iteration flow control

["Gyula Fora (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gyula Fora closed FLINK-2497.
-----------------------------
    Resolution: Won't Fix

> Rework streaming iteration flow control
> ---------------------------------------
>
>                 Key: FLINK-2497
>                 URL: https://issues.apache.org/jira/browse/FLINK-2497
>             Project: Flink
>          Issue Type: Improvement
>          Components: API / DataStream
>            Reporter: Gyula Fora
>            Priority: Minor
>              Labels: auto-deprioritized-major
>
> Currently streaming iterations use a simple Blocking queue to pass data from the iteration tails to the iteration heads.
> While this works most of the time, during increased load (or even short bursts) the increased amount of records in transit inside the loop can cause the limited number of input/output buffer-pools to fill up and deadlock the system. (caused by the back-pressure mechanism)
> One solution to this problem could be to remove back-pressure on the loop-edge by replacing the Blocking queue with some soft spilling queue that will write the records automatically to disk letting the iteration tail to progress with its input.
> For a more robust solution we need to implement more elaborate flow control in streaming programs containing iterations.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)