link on yarn error “Yarn only has -1 virtual cores available”

[张炜 <zh...@gmail.com>]

Dear all,

I have installed Cloudera CDH 6, and then I want to install flink using
package from flink website, and run it "flink on yarn".

I have down the following steps: 1. edit /etc/profile

export
HADOOP_CONF_DIR=/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hadoop/etc/hadoop

   1. execute ./yarn-session.sh -n 4 -jm 2048 -tm 2048 -s 3 -nm
   FlinkOnYarnSession -d -st

But it always shows：

The number of virtual cores per node were configured with 4 but Yarn only
has -1 virtual cores available. Please note that the number of virtual
cores is set to the number of task slots by default unless configured in
the Flink config with 'yarn.containers.vcores.'

This is a new cluster and there is no job running, and I see in yarn webUI
there is 20 vcores available.

Also after I tried to add yarn.containers.vcores in flink-conf.yml, say I
add a line yarn.containers.vcores: 8, when I run the yarn-session command,
it will throw an error "The number of virtual cores per node were
configured with 8 but Yarn only has -1 virtual cores available.".
[image: flink-error.png]

Please help with this problem. Thank you very much!

Re: link on yarn error “Yarn only has -1 virtual cores available”

[Dawid Wysakowicz <dw...@apache.org>]

Hi,

It sounds like this problem[1], which will be fixed for flink 1.5.5,
1.6.2, 1.7.0. The problem is flink reads local config file which might
not reflect the correct setting of the yarn cluster.

You may try setting the yarn parameter
"yarn.nodemanager.resource.cpu-vcores" to the max number of vcores per
node in your cluster.

Best,

Dawid


[1] https://issues.apache.org/jira/browse/FLINK-5542

On 23/10/2018 06:52, 张炜 wrote:
> Dear all,
>
> I have installed Cloudera CDH 6, and then I want to install flink
> using package from flink website, and run it "flink on yarn".
>
> I have down the following steps: 1. edit /etc/profile
>
> export
> HADOOP_CONF_DIR=/opt/cloudera/parcels/CDH-6.0.0-1.cdh6.0.0.p0.537114/lib/hadoop/etc/hadoop
>
>  2. execute ./yarn-session.sh -n 4 -jm 2048 -tm 2048 -s 3 -nm
>     FlinkOnYarnSession -d -st
>
> But it always shows：
>
> The number of virtual cores per node were configured with 4 but Yarn
> only has -1 virtual cores available. Please note that the number of
> virtual cores is set to the number of task slots by default unless
> configured in the Flink config with 'yarn.containers.vcores.'
>
> This is a new cluster and there is no job running, and I see in yarn
> webUI there is 20 vcores available.
>
> Also after I tried to add yarn.containers.vcores in flink-conf.yml,
> say I add a line yarn.containers.vcores: 8, when I run the
> yarn-session command, it will throw an error "The number of virtual
> cores per node were configured with 8 but Yarn only has -1 virtual
> cores available.".
>
> flink-error.png
>
> Please help with this problem. Thank you very much!
>
>