You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Piotr Nowojski (Jira)" <ji...@apache.org> on 2022/07/11 14:52:00 UTC
[jira] [Commented] (FLINK-22643) Too many TCP connections among TaskManagers for large scale jobs
[ https://issues.apache.org/jira/browse/FLINK-22643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565043#comment-17565043 ]
Piotr Nowojski commented on FLINK-22643:
----------------------------------------
After closing a ticket I forgot to mention that this seemed to have improved low latency performance of the network stack by quite a bit !Screenshot 2022-07-11 at 16.49.42.png! Previously the difference between 100ms and 1ms in the microbenchmark was ~18%, after merging this change it's around ~4%.
> Too many TCP connections among TaskManagers for large scale jobs
> ----------------------------------------------------------------
>
> Key: FLINK-22643
> URL: https://issues.apache.org/jira/browse/FLINK-22643
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Affects Versions: 1.14.0, 1.13.2
> Reporter: Zhilong Hong
> Assignee: fanrui
> Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
> Fix For: 1.15.0
>
> Attachments: Screenshot 2022-07-11 at 16.49.42.png
>
>
> For the large scale jobs, there will be too many TCP connections among TaskManagers. Let's take an example.
> For a streaming job with 20 JobVertices, each JobVertex has 500 parallelism. We divide the vertices into 5 slot sharing groups. Each TaskManager has 5 slots. Thus there will be 400 taskmanagers in this job. Let's assume that job runs on a cluster with 20 machines.
> If all the job edges are all-to-all edges, there will be 19 * 20 * 399 * 2 = 303,240 TCP connections for each machine. If we run several jobs on this cluster, the TCP connections may exceed the maximum limit of linux, which is 1,048,576. This will stop the TaskManagers from creating new TCP connections and cause task failovers.
> As we run our production jobs on a K8S cluster, the job always failover due to exceptions related to network, such as {{Sending the partition request to 'null' failed}}, and etc.
> We think that we can decrease the number of connections by letting tasks reuse the same connection. We implemented a POC that makes all tasks on the same TaskManager reuse one TCP connection. For the example job we mentioned above, the number of connections will decrease from 303,240 to 15960. With the POC, the frequency of meeting exceptions related to network in our production jobs drops significantly.
> The POC is illustrated in: https://github.com/wsry/flink/commit/bf1c09e80450f40d018a1d1d4fe3dfd2de777fdc
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)