[jira] [Commented] (FLINK-10462) Remove ConnectionIndex for further sharing tcp connection in credit-based mode

["Nico Kruber (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16633879#comment-16633879 ] 

Nico Kruber commented on FLINK-10462:
-------------------------------------

Yes, you are right about the changes that credit-based flow control allows here. I have been thinking about this as well and have also been asked at Flink Forward, but I have mixed feelings: sometimes it is actually beneficial to have multiple TCP connections if, for example, network is the bottleneck. Only through multiple connections, you will be able to saturate a connection, judging from various network hardware tests I have read in the past. This especially happens the more throughput the interface offers.

Have you made different observations?

On the other hand, opening too many connections (happens easily for large topologies) may also become a problem. How about a more generic connection pool with configurable size? This wouldn't be an easy patch as the ticket described here and we'd also only need to look into this once we really have the problem with too many connections, or do you see any other advantages of changing the behaviour here?

> Remove ConnectionIndex for further sharing tcp connection in credit-based mode 
> -------------------------------------------------------------------------------
>
>                 Key: FLINK-10462
>                 URL: https://issues.apache.org/jira/browse/FLINK-10462
>             Project: Flink
>          Issue Type: Improvement
>          Components: Network
>    Affects Versions: 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.6.1, 1.5.4
>            Reporter: zhijiang
>            Assignee: zhijiang
>            Priority: Minor
>
> Every {{IntermediateResult}} generates a random {{ConnectionIndex}} which will be included in {{ConnectionID}}.
> The {{RemoteInputChannel}} requests to establish tcp connection via {{ConnectionID}}. That means one tcp connection may be shared by multiple {{RemoteInputChannel}} {{s which have the same ConnectionID}}. To do so, we can reduce the physical connections between two \{{TaskManager}} s, and it brings benefits for large scale jobs. 
> But this sharing is limited only for the same {{IntermediateResult}}, and I think it is mainly because we may temporarily switch off {{autoread}} for the channel during back pressure in previous network flow control. For credit-based mode, the channel is always open for transporting different intermediate data, so we can further share the tcp connection for different {{IntermediateResults}} to remove the limit. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)