[jira] [Closed] (FLINK-1703) option to balance load according to CPU available rather than slots available

["Greg Hogan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Hogan closed FLINK-1703.
-----------------------------
    Resolution: Duplicate

> option to balance load according to CPU available rather than slots available
> -----------------------------------------------------------------------------
>
>                 Key: FLINK-1703
>                 URL: https://issues.apache.org/jira/browse/FLINK-1703
>             Project: Flink
>          Issue Type: Improvement
>          Components: Streaming
>            Reporter: Emmanuel Leroy
>              Labels: balancer, load
>
> To be able to accommodate for loading and running several different topologies on a cluster, one may want to assign more slots per node than there are CPUs available (many slots per CPU).
>  
> For example, with 3 nodes with 3 CPUs each, defining 3 slots per node (9 slots total) limits the number of task that can be loaded at any one time. 
> => Increasing the number of slots to 3 per CPU, gives 27 slots, allowing to load more topologies.
> In this configuration (3 nodes, 3 CPUs, 27 slots), if a job specifies a parallelism of 9, 9 slots out of 27 are used, but the distribution strategy gives preference to slots on the same host as the stream source to limit network IOs: as a consequence, the first job using 9 slots has the 9 slots allocated on a single host (the one on the same host as the stream source), even though this host only has 3 CPUs available. 
> These 9 slots are therefore sharing the same 3 CPUs, while the other 2 nodes (6 CPUs) stay idle.
> The distribution strategy should take into account the number of CPUs available along with the number of slots rather than rely only on the slot availability.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)