You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by qq <47...@qq.com> on 2019/11/14 07:22:59 UTC
Flink on Yarn resource arrangement
Hi all,
Could you list details how Flink job on Yarn resources managed ?
I used command “-p 20 -yn 5 -ys 3 -yjm 2048m -ytm 2048m” to run flink job. I got
containers vcores
8 22
Task Managers 7 Total Task Slots 21
I used command “-p 20 -yn 7 -ys 4 -yjm 2048m -ytm 2048m” to run flink job, I got
containers vcores
6 21
Total Task Slots 20 Task Managers 5
Could you help give the exactly resources formula ? Thanks very much.
Alex Fu
2019/11/14
Re: Flink on Yarn resource arrangement
Posted by vino yang <ya...@gmail.com>.
Hi Alex,
Which Flink version are you using?
AFAIK, since Flink 1.8+, the config option: "-yn" for Flink on YARN job
cluster mode does not take effect(always 1 and would be overridden).
So, the config option "-ys" and "-p" will decide the number of TM.
The first example: -p(20)/-ys(3) should be advanced to 7 to match the
requirement of parallelism. So total slots should be 7 * 3 = 21.
The second example: -p(20)/-ys(4) = 5. So total slots should be 5 * 4 = 20.
Best,
Vino
qq <47...@qq.com> 于2019年11月14日周四 下午3:26写道：
> Hi all,
>
> Could you list details how Flink job on Yarn resources managed ?
>
> I used command “-p 20 -yn 5 -ys 3 -yjm 2048m -ytm 2048m” to run flink
> job. I got
> containers vcores
> 8 22
> Task Managers 7 Total Task Slots 21
>
>
> I used command “-p 20 -yn 7 -ys 4 -yjm 2048m -ytm 2048m” to run flink job,
> I got
> containers vcores
> 6 21
> Total Task Slots 20 Task Managers 5
>
> Could you help give the exactly resources formula ? Thanks very much.
>
>
>
> Alex Fu
> 2019/11/14
Re: Flink on Yarn resource arrangement
Posted by vino yang <ya...@gmail.com>.
Hi Alex,
Which Flink version are you using?
AFAIK, since Flink 1.8+, the config option: "-yn" for Flink on YARN job
cluster mode does not take effect(always 1 and would be overridden).
So, the config option "-ys" and "-p" will decide the number of TM.
The first example: -p(20)/-ys(3) should be advanced to 7 to match the
requirement of parallelism. So total slots should be 7 * 3 = 21.
The second example: -p(20)/-ys(4) = 5. So total slots should be 5 * 4 = 20.
Best,
Vino
qq <47...@qq.com> 于2019年11月14日周四 下午3:26写道：
> Hi all,
>
> Could you list details how Flink job on Yarn resources managed ?
>
> I used command “-p 20 -yn 5 -ys 3 -yjm 2048m -ytm 2048m” to run flink
> job. I got
> containers vcores
> 8 22
> Task Managers 7 Total Task Slots 21
>
>
> I used command “-p 20 -yn 7 -ys 4 -yjm 2048m -ytm 2048m” to run flink job,
> I got
> containers vcores
> 6 21
> Total Task Slots 20 Task Managers 5
>
> Could you help give the exactly resources formula ? Thanks very much.
>
>
>
> Alex Fu
> 2019/11/14