You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Piotr Nowojski (JIRA)" <ji...@apache.org> on 2018/12/12 12:10:00 UTC

[jira] [Commented] (FLINK-11035) Notify data available to network stack immediately after finishing BufferBuilder

    [ https://issues.apache.org/jira/browse/FLINK-11035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718897#comment-16718897 ] 

Piotr Nowojski commented on FLINK-11035:
----------------------------------------

[~zjwang] Have you observed this to be a problem in the real world? Problem is that any kind of notifications in this area can be quite costly, however now I'm not sure if I was benchmarking fix for that or not (I would guess yes, but again - I'm not sure). When I was implementing this I was aware about this scenario that you are describing, but it seemed like a non-issue - we can always relay on output flushing to do the trick. After all this only affects scenarios where:
1. we are not under full throughput, because we are blocked on requesting next buffer
2. it only increases the latency for records in rare cases when buffer is full after serialising next record. In other words, if record i 320bytes long, this extra notification will only help for 1 record out of 100.

> Notify data available to network stack immediately after finishing BufferBuilder
> --------------------------------------------------------------------------------
>
>                 Key: FLINK-11035
>                 URL: https://issues.apache.org/jira/browse/FLINK-11035
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Network
>    Affects Versions: 1.8.0
>            Reporter: zhijiang
>            Assignee: zhijiang
>            Priority: Minor
>
> The data availability notification for network relies on whether there are finished _BufferBuilder_ or flush triggered. If flush is not triggered and the first _BufferBuilder_ enqueues into the subpartition, although this _BufferBuilder_ is finished on _RecordWriter_ side, it has to rely on enqueuing the second _BufferBuilder_ to trigger notification available.  It may bring some delays for transporting the finished _BufferBuilder_ in network, especially there has a blocking operation for requesting the second _BufferBuilder_ from pool.
> Supposing there is only one available buffer in LocalBufferPool in extreme scenarios, if the first _BufferBuilder_ is not transported and recycled, the requesting for second _BufferBuilder_ will be blocked all the time.
> I propose to add a _notifyBufferFinished_ method in _ResultPartitionWriter_ interface, then _RecordWriter_ can notify via it after _BufferBuilder_ finished_._
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)