You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Jiang Xin (Jira)" <ji...@apache.org> on 2023/11/28 03:09:00 UTC

[jira] [Created] (FLINK-33668) Decoupling Shuffle network memory and job topology

Jiang Xin created FLINK-33668:
---------------------------------

             Summary: Decoupling Shuffle network memory and job topology
                 Key: FLINK-33668
                 URL: https://issues.apache.org/jira/browse/FLINK-33668
             Project: Flink
          Issue Type: Improvement
          Components: Runtime / Network
            Reporter: Jiang Xin
             Fix For: 1.19.0


With [FLINK-30469|https://issues.apache.org/jira/browse/FLINK-30469]  and [FLINK-31643|https://issues.apache.org/jira/browse/FLINK-31643], we have decoupled the shuffle network memory and the parallelism of tasks by limiting the number of buffers for each InputGate and ResultPartition. However, when too many shuffle tasks are running simultaneously on the same TaskManager, "Insufficient number of network buffers" errors would still occur. This usually happens when Slot Sharing Group is enabled or a TaskManager contains multiple slots.

So we need to make sure that the TaskManager does not encounter "Insufficient number of network buffers" even if there are dozens of InputGates and ResultPartitions running on the same TaskManager simultaneously.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)