[jira] [Commented] (FLINK-18799) improve slot allocation to make resource balance among machines.

["Zhu Zhu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-18799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169755#comment-17169755 ] 

Zhu Zhu commented on FLINK-18799:
---------------------------------

If your jobs are DataStream jobs, you can assign job vertices to different slot sharing groups to achieve this goal. You can also set slot sharing groups of operators to be null (e.g. {{map.slotSharingGroup(null)}}) so that Flink will do this automatically for you. Each subtask will occupy one slot in this way so you also need to increase the slot number of taksmanagers.

> improve slot allocation to make resource balance among machines.
> ----------------------------------------------------------------
>
>                 Key: FLINK-18799
>                 URL: https://issues.apache.org/jira/browse/FLINK-18799
>             Project: Flink
>          Issue Type: Improvement
>          Components: API / Core, Client / Job Submission
>            Reporter: nobleyd
>            Priority: Major
>
>   I have a completed job, and each vertex may have different parallelism, and what troubles me is that the metric 'cpu used' differs among machines.
>   It comes to be good when I upgraded to use flink1.10, and add 'cluster.evenly-spread-out-slots: true' to flink config. This is good, while sometimes it is not enough.
>   For example, I have 5 taskmanagers(each deployed in one machine). I have a job, and some vertexs and the parallelism info is below.
>  
> ||vertex||parallelism||
> |A|1|
> |B|15|
> |C|20|
> |D|1|
> |E|15|
> In this case, the resource sometimes won't balance very good. What I expected is that the vertext B/C/E can distribute evenly amont 5 taskmanagers. Vertex A and D only have 1 parallelism, and it is just some config stream. 
>   Expected allocation strategy: For each vertex, allocate slot evenly among taskmanagers. Then next vertex and repeat. For example, the result below:
>  
>  
> ||TaskManager1||TaskManager2||TaskManager3||TaskManager4||TaskManager5||
> |{color:#FF0000}A1{color}|{color:#00875a}B1{color}|{color:#00875a}B2{color}|{color:#00875a}B3{color}|{color:#00875a}B4{color}|
> |{color:#00875a}B5{color}|{color:#00875a}B6{color}|{color:#00875a}B7{color}|{color:#00875a}B8{color}|{color:#00875a}B9{color}|
> |{color:#00875a}B10{color}|{color:#00875a}B11{color}|{color:#00875a}B12{color}|{color:#00875a}B13{color}|{color:#00875a}B14{color}|
> |{color:#00875a}B15{color}|{color:#ff8b00}C1{color}|{color:#ff8b00}C2{color}|{color:#ff8b00}C3{color}|{color:#ff8b00}C4{color}|
> |{color:#ff8b00}C5{color}|{color:#ff8b00}C6{color}|{color:#ff8b00}C7{color}|{color:#ff8b00}C8{color}|{color:#ff8b00}C9{color}|
> |{color:#ff8b00}C10{color}|{color:#ff8b00}C11{color}|{color:#ff8b00}C12{color}|{color:#ff8b00}C13{color}|{color:#ff8b00}C14{color}|
> |{color:#ff8b00}C15{color}|{color:#ff8b00}C16{color}|{color:#ff8b00}C17{color}|{color:#ff8b00}C18{color}|{color:#ff8b00}C19{color}|
> |{color:#ff8b00}C20{color}|{color:#403294}D1{color}|E1|E2|E3|
> |E4|E5|E6|E7|E8|
> |E9|E10|E11|E12|E13|
> |E14|E15| | | |
> | | | | | |
> The allocation order is A -> B -> C -> D -> E or some other order, it doesn't matter. The key point is one vertex's all  parallel subtasks should be allocated at one time, and then to consider the next vertex. With this strategy, vertext A/D won't influence other vertex's distribution equilibrium.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)