[jira] [Created] (FLINK-3164) Spread out scheduling strategy

["Greg Hogan (JIRA)" <ji...@apache.org>]

Greg Hogan created FLINK-3164:
---------------------------------

             Summary: Spread out scheduling strategy
                 Key: FLINK-3164
                 URL: https://issues.apache.org/jira/browse/FLINK-3164
             Project: Flink
          Issue Type: Improvement
          Components: Distributed Runtime, Java API, Scala API
    Affects Versions: 1.0.0
            Reporter: Greg Hogan


The size of a Flink cluster is bounded by the amount of memory allocated for network buffers. The all-to-all distribution of data during a network shuffle means that doubling the number of TaskManager slots quadruples the required number of network buffers.

A Flink job can be configured to execute operators with lower parallelism which reduces the number of network buffers used across the cluster. Since the Flink scheduler clusters tasks the number of network buffers to be configured cannot be reduced.

For example, if each TaskManager has 32 slots and the cluster has 32 TaskManagers the maximum parallelism can be set to 1024. If the preceding operator has a parallelism of 32 then the TaskManager fan-out is between 1*1024 (tasks evenly distributed) and 32*1024 (executed on a single TaskManager).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)