[jira] [Commented] (FLINK-11082) Increase backlog only if it is available for consumption

["Piotr Nowojski (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16732053#comment-16732053 ] 

Piotr Nowojski commented on FLINK-11082:
----------------------------------------

> Furthermore, the backlog would affect requesting floating buffers on downstream side. That means some floating buffers are fetched in advance but not be used for long time, so the floating buffers are not made use of efficiently.

I think that wouldn't be an issue, if floating vs exclusive were abstract concept. Like if we assign a floating credit to a channel (for example bumping number of assigned buffers from 2 up to 3), once we release ANY buffer from this channel and the number goes back down to "2", it could be interpreted/be an equivalent of saying that the channel released floating buffer and currently holds only to two exclusive buffers. As far as I remember, currently that's not the case, right? If we have assigned 2 exclusive buffers, then we assign one floating buffer, floating buffer is released only we process all three of those buffers, right?

Maybe that's the root problem that we should fix? I think I vaguely remember this same behaviour causing some other problems.

> Increase backlog only if it is available for consumption
> --------------------------------------------------------
>
>                 Key: FLINK-11082
>                 URL: https://issues.apache.org/jira/browse/FLINK-11082
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Network
>    Affects Versions: 1.8.0
>            Reporter: zhijiang
>            Assignee: zhijiang
>            Priority: Minor
>
> The backlog should indicate how many buffers are available in subpartition for downstream's  consumption. The availability is considered from two factors. One is {{BufferConsumer}} finished, and the other is flush triggered.
> In current implementation, when the {{BufferConsumer}} is added into the subpartition, then the backlog is increased as a result, but this {{BufferConsumer}} is not yet available for network transport.
> Furthermore, the backlog would affect requesting floating buffers on downstream side. That means some floating buffers are fetched in advance but not be used for long time, so the floating buffers are not made use of efficiently.
> We found this scenario extremely for rebalance selector on upstream side, so we want to change when to increase backlog by finishing {{BufferConsumer}} or flush triggered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)