[jira] [Issue Comment Deleted] (FLINK-16645) Limit the maximum backlogs in subpartitions for data skew case

["Jiayi Liao (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jiayi Liao updated FLINK-16645:
-------------------------------
    Comment: was deleted

(was: [~zjwang] Okay. I think this option should only work in {{ChannelSelectorRecordWriter}}, right?  )

> Limit the maximum backlogs in subpartitions for data skew case
> --------------------------------------------------------------
>
>                 Key: FLINK-16645
>                 URL: https://issues.apache.org/jira/browse/FLINK-16645
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Network
>            Reporter: Zhijiang
>            Assignee: Jiayi Liao
>            Priority: Major
>             Fix For: 1.11.0
>
>
> In the case of data skew, most of the buffers in partition's LocalBufferPool are probably requested away and accumulated in certain subpartition, which would increase in-flight data to slow down the barrier alignment.
> We can set up a proper config to control how many backlogs are allowed for one subpartition. If one subpartition reaches this threshold, it will make the buffer pool unavailable which blocks task processing continuously. Then we can reduce the in-flight data for speeding up checkpoint process a bit and not impact on the performance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)