[jira] [Created] (FLINK-1703) option to balance load according to CPU available rather than slots available

["Emmanuel Leroy (JIRA)" <ji...@apache.org>]

Emmanuel Leroy created FLINK-1703:
-------------------------------------

             Summary: option to balance load according to CPU available rather than slots available
                 Key: FLINK-1703
                 URL: https://issues.apache.org/jira/browse/FLINK-1703
             Project: Flink
          Issue Type: Improvement
          Components: Streaming
            Reporter: Emmanuel Leroy


To be able to accommodate for loading and running several different topologies on a cluster, one may want to assign more slots per node than there are CPUs available (many slots per CPU).
 
For example, with 3 nodes with 3 CPUs each, defining 3 slots per node (9 slots total) limits the number of task that can be loaded at any one time. 
=> Increasing the number of slots to 3 per CPU, gives 27 slots, allowing to load more topologies.

In this configuration (3 nodes, 3 CPUs, 27 slots), if a job specifies a parallelism of 9, 9 slots out of 27 are used, but the distribution strategy gives preference to slots on the same host as the stream source to limit network IOs: as a consequence, the first job using 9 slots has the 9 slots allocated on a single host (the one on the same host as the stream source), even though this host only has 3 CPUs available. 
These 9 slots are therefore sharing the same 3 CPUs, while the other 2 nodes (6 CPUs) stay idle.

The distribution strategy should take into account the number of CPUs available along with the number of slots rather than rely only on the slot availability.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)