You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "ani.desh1512" <an...@gmail.com> on 2017/01/24 23:12:25 UTC
Flink with Yarn on MapR
Hi,
I am trying to setup flink with Yarn on Mapr cluster. I built flink
(flink-1.3-SNAPSHOT) as follows:
mvn clean install -DskipTests -Pvendor-repos
-Dhadoop.version=2.7.0-mapr-1607
The build is successful. Then I try to run ./bin/yarn-session.sh -n 4
(without changing any config or whatsoever) and get the following error:
Error while deploying YARN cluster: Couldn't deploy Yarn cluster
java.lang.RuntimeException: Couldn't deploy Yarn cluster
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:425)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:620)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:476)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:473)
at
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native
Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:473)
Caused by: java.lang.NumberFormatException: For input string:
"${nodemanager.resource.cpu-vcores}"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:569)
at java.lang.Integer.parseInt(Integer.java:615)
at
org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1271)
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.isReadyForDeployment(AbstractYarnClusterDescriptor.java:315)
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:434)
at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:423)
... 9 more
Now, the property that is causing this error nodemanager.resource.cpu-vcores
is appropriately set in yarn-site.xml. The cluster is 3 ResourceManager (2
on standby) and 5 NodeManager. To be extra safe, I changed the value for
this property at ALL the Nodemanager’s yarn-site.xml.
I believe that this property is default set to 4. So I am trying to
understand as to why is this error cropping up.
The required environment variable is set as follows:
YARN_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/
I also tried setting the fs.hdfs.hadoopconf property (to point to the Hadoop
conf directory) in flink-config.yaml. But I still get the same error.
Any help with this error will be greatly appreciated
Thanks in advance,
Aniket D
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-with-Yarn-on-MapR-tp11275.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Flink with Yarn on MapR
Posted by Robert Metzger <rm...@apache.org>.
Hi,
I think this is a re-post of a question I've already "answered":
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-with-Yarn-on-MapR-td15448.html
On Wed, Jan 25, 2017 at 12:12 AM, ani.desh1512 <an...@gmail.com>
wrote:
> Hi,
> I am trying to setup flink with Yarn on Mapr cluster. I built flink
> (flink-1.3-SNAPSHOT) as follows:
>
> mvn clean install -DskipTests -Pvendor-repos
> -Dhadoop.version=2.7.0-mapr-1607
>
> The build is successful. Then I try to run ./bin/yarn-session.sh -n 4
> (without changing any config or whatsoever) and get the following error:
>
> Error while deploying YARN cluster: Couldn't deploy Yarn cluster
> java.lang.RuntimeException: Couldn't deploy Yarn cluster
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(
> AbstractYarnClusterDescriptor.java:425)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(
> FlinkYarnSessionCli.java:620)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(
> FlinkYarnSessionCli.java:476)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(
> FlinkYarnSessionCli.java:473)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(
> HadoopSecurityContext.java:43)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1595)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(
> HadoopSecurityContext.java:40)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(
> FlinkYarnSessionCli.java:473)
> Caused by: java.lang.NumberFormatException: For input string:
> "${nodemanager.resource.cpu-vcores}"
> at
> java.lang.NumberFormatException.forInputString(
> NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:569)
> at java.lang.Integer.parseInt(Integer.java:615)
> at
> org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1271)
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.isReadyForDeployment(
> AbstractYarnClusterDescriptor.java:315)
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(
> AbstractYarnClusterDescriptor.java:434)
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(
> AbstractYarnClusterDescriptor.java:423)
> ... 9 more
>
> Now, the property that is causing this error nodemanager.resource.cpu-
> vcores
> is appropriately set in yarn-site.xml. The cluster is 3 ResourceManager (2
> on standby) and 5 NodeManager. To be extra safe, I changed the value for
> this property at ALL the Nodemanager’s yarn-site.xml.
> I believe that this property is default set to 4. So I am trying to
> understand as to why is this error cropping up.
> The required environment variable is set as follows:
> YARN_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/
>
> I also tried setting the fs.hdfs.hadoopconf property (to point to the
> Hadoop
> conf directory) in flink-config.yaml. But I still get the same error.
>
>
> Any help with this error will be greatly appreciated
>
>
> Thanks in advance,
>
> Aniket D
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Flink-with-Yarn-
> on-MapR-tp11275.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>