You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by 徐涛 <ha...@gmail.com> on 2019/02/18 02:49:24 UTC

Each yarn container only use 1 vcore even if taskmanager.numberOfTaskSlots is set

Hi Experts,
	I am running Flink 1.7.1 program on Yarn 2.7, the taskmanager.numberOfTaskSlots is set to 4. The parallelism.default is set to 8. When the program is running, 3 yarn containers is launched, but each of them only use 1 vcore, I think by default the number of vcores is set to the number of slots per TaskManager.
	Because it can not get enough CPU resources, the CPU load may be high. So how can this problem be fixed? 
	Thanks a lot for your help.

Best
Henry

Re: Each yarn container only use 1 vcore even if taskmanager.numberOfTaskSlots is set

Posted by Gary Yao <ga...@ververica.com>.

Hi Henry,

If I understand you correctly, you want YARN to allocate 4 vcores per TM
container. You can achieve this by enabling the FairScheduler in YARN
[1][2].

Best,
Gary

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#yarn-containers-vcores
[2]
https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/FairScheduler.html

On Mon, Feb 18, 2019 at 3:49 AM 徐涛 <ha...@gmail.com> wrote:

> Hi Experts,
>         I am running Flink 1.7.1 program on Yarn 2.7, the
> taskmanager.numberOfTaskSlots is set to 4. The parallelism.default is set
> to 8. When the program is running, 3 yarn containers is launched, but each
> of them only use 1 vcore, I think by default the number of vcores is set to
> the number of slots per TaskManager.
>         Because it can not get enough CPU resources, the CPU load may be
> high. So how can this problem be fixed?
>         Thanks a lot for your help.
>
> Best
> Henry