You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Piotr Nowojski (Jira)" <ji...@apache.org> on 2020/10/14 07:26:00 UTC
[jira] [Closed] (FLINK-14396) Implement rudimentary non-blocking
network output
[ https://issues.apache.org/jira/browse/FLINK-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Piotr Nowojski closed FLINK-14396.
----------------------------------
> Implement rudimentary non-blocking network output
> -------------------------------------------------
>
> Key: FLINK-14396
> URL: https://issues.apache.org/jira/browse/FLINK-14396
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
> Reporter: Zhijiang
> Assignee: Zhijiang
> Priority: Minor
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Considering the mailbox model and unaligned checkpoints requirements in future, task network output should be non-blocking. In other words, as long as output is available, it should never block for a subsequent/future single record write.
> In the first version, we only implement the non-blocking output for the most regular case, and do not solve the following cases which still keep the previous behavior.
> * Big record which might span multiple buffers
> * Flatmap-like operators which might emit multiple records in every process
> * Broadcast watermark which might request multiple buffers at a time
> The solution is providing the RecordWriter#isAvailable method and respective LocalBufferPool#isAvailable for judging the output beforehand. As long as there is at-least one available buffer in LocalBufferPool, the RecordWriter is available for network output in most cases. This doesn’t include runtime handling of this non-blocking and availability behavior in StreamInputProcessor
> Note: It requires the minimum number of buffers in output LocalBufferPool adjusting to (numberOfSubpartitions + 1) and also adjusting the monitor of backpressure future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)