You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Zhijiang (Jira)" <ji...@apache.org> on 2020/07/21 07:17:00 UTC
[jira] [Closed] (FLINK-11037) Introduce another greedy mechanism
for distributing floating buffers
[ https://issues.apache.org/jira/browse/FLINK-11037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhijiang closed FLINK-11037.
----------------------------
Resolution: Later
I guess we do not have much time to further work on it recently as there are no evidence for problems ATM. I will close it for cleanup until we find it necessary to reopen future.
> Introduce another greedy mechanism for distributing floating buffers
> --------------------------------------------------------------------
>
> Key: FLINK-11037
> URL: https://issues.apache.org/jira/browse/FLINK-11037
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Affects Versions: 1.8.0
> Reporter: Zhijiang
> Assignee: Zhijiang
> Priority: Minor
>
> The current mechanism for distributing floating buffers is fair for all the listeners. In detail, each input channel can only request one floating buffer each time although this channel may actually need more floating buffers. Then this channel has to loop to request floating buffer until all are satisfied or pool is exhausted.
> In generally speaking, this way seems fair for all the concurrent channels invoked by netty nio thread. But every request from LocalBufferPool needs to syn lock and it is hard to say how to distribute all the available floating buffers behaves better in real scenarios.
> Therefore we propose another greedy mechanism to request more floating buffers each time. In extreme case, we can even request all the required buffers at a time or partial ones via configured parameters. On the other side, LocalBufferPool can also decide how many floating buffers should been assigned based on some factors, such as how many total channels and how many total floating buffers.
> The motivation is making better use of floating buffer resources and it may need extra metrics for adjusting the mechanism dynamically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)