[jira] [Commented] (FLINK-10462) Remove ConnectionIndex for further sharing tcp connection in credit-based mode

["Yingjie Cao (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006711#comment-17006711 ] 

Yingjie Cao commented on FLINK-10462:
-------------------------------------

Recently, we encountered the same problem under different scenario and we found connection reuse is important for short queries. I create a new Jira  FLINK-15455 which is related to this one. 

To restrict the number of connections, I think we can add a new option which indicates the maximum number of connections between two Taskmanagers and then we map the connectionIndex generated by IntermediateResult to a new value smaller than the maximum connection number, for example, we can do like this:
{code:java}
newConnectionIndex = oldConnectionIndex % maxNumConnections{code}
Everything else can keep unchanged and we can set the default value of maxNumConnections to Integer.MAX_VALUE which can keep the behavior of network connection unchanged so we don't need to worry about any potential regressions. If the users find that there are too many connections, then he/she can set the maxNumConnections to a relative smaller value.

What do you think?

> Remove ConnectionIndex for further sharing tcp connection in credit-based mode 
> -------------------------------------------------------------------------------
>
>                 Key: FLINK-10462
>                 URL: https://issues.apache.org/jira/browse/FLINK-10462
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Network
>    Affects Versions: 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.6.0, 1.6.1
>            Reporter: zhijiang
>            Assignee: zhijiang
>            Priority: Minor
>
> Every {{IntermediateResult}} generates a random {{ConnectionIndex}} which will be included in {{ConnectionID}}.
> The {{RemoteInputChannel}} requests to establish tcp connection via {{ConnectionID}}. That means one tcp connection may be shared by multiple {{RemoteInputChannel}} {{s which have the same ConnectionID}}. To do so, we can reduce the physical connections between two \{{TaskManager}} s, and it brings benefits for large scale jobs. 
> But this sharing is limited only for the same {{IntermediateResult}}, and I think it is mainly because we may temporarily switch off {{autoread}} for the channel during back pressure in previous network flow control. For credit-based mode, the channel is always open for transporting different intermediate data, so we can further share the tcp connection for different {{IntermediateResults}} to remove the limit. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)