You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Nishu <ni...@gmail.com> on 2017/11/20 12:53:16 UTC
Flink session on yarn
Hi,
I am trying to start flink session(v1.3.2) on yarn(v 2.7) on HDInsight
cluster. But it throws following error:
*Error while deploying YARN cluster: Couldn't deploy Yarn cluster*
*java.lang.RuntimeException: Couldn't deploy Yarn cluster*
* at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:443)*
* at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:630)*
* at
org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:486)*
* at
org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:483)*
* at
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)*
* at java.security.AccessController.doPrivileged(Native Method)*
* at javax.security.auth.Subject.doAs(Subject.java:422)*
* at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)*
* at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)*
* at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:483)*
*Caused by: java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException: Class
org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not
found*
* at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)*
* at
org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:161)*
* at
org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94)*
* at
org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)*
I referred this mail thread also:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-session-on-Yarn-ClassNotFoundException-td15222.html
I am using HDInsight Cluster(HDP3.6). According to the documentation, set
HADOOP_CONF_DIR and YARN_CONF_DIR as well.
Any inputs will be really helpful. Thanks!
--
Thanks & Regards,
Nishu Tayal
Re: Flink session on yarn
Posted by Thomas Nadler <th...@tngtech.com>.
Hi,
instead of using the RequestHedgingRMFailoverProxyProvider you could try
to use the org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.
You can configure this in the YARN configs =>
yarn.client.failover-proxy-provider.
Kind regards,
Thomas
On 11/20/2017 01:53 PM, Nishu wrote:
> Hi,
>
> I am trying to start flink session(v1.3.2) on yarn(v 2.7) on HDInsight
> cluster. But it throws following error:
>
> *Error while deploying YARN cluster: Couldn't deploy Yarn cluster*
> *java.lang.RuntimeException: Couldn't deploy Yarn cluster*
> * at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:443)*
> * at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:630)*
> * at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:486)*
> * at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:483)*
> * at
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)*
> * at java.security.AccessController.doPrivileged(Native Method)*
> * at javax.security.auth.Subject.doAs(Subject.java:422)*
> * at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)*
> * at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)*
> * at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:483)*
> *Caused by: java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider
> not found*
> * at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)*
> * at
> org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:161)*
> * at
> org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94)*
> * at
> org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)*
>
>
> I referred this mail thread
> also: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-session-on-Yarn-ClassNotFoundException-td15222.html
>
> I am using HDInsight Cluster(HDP3.6). According to the
> documentation, set HADOOP_CONF_DIR and YARN_CONF_DIR as well.
>
> Any inputs will be really helpful. Thanks!
>
> --
> Thanks & Regards,
> Nishu Tayal
--
Thomas Nadler * thomas.nadler@tngtech.com * +49-176-17654608
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Müller
Amtsgericht München, HRB 135082
Re: Flink session on yarn
Posted by Federico D'Ambrosio <fe...@gmail.com>.
Hi Nishu,
did you compile Flink from sources as recommended here [1]?
Regards,
Federico
[1] https://ci.apache.org/projects/flink/flink-docs-
release-1.3/setup/building.html#vendor-specific-versions
2017-11-20 13:53 GMT+01:00 Nishu <ni...@gmail.com>:
> Hi,
>
> I am trying to start flink session(v1.3.2) on yarn(v 2.7) on HDInsight
> cluster. But it throws following error:
>
> *Error while deploying YARN cluster: Couldn't deploy Yarn cluster*
> *java.lang.RuntimeException: Couldn't deploy Yarn cluster*
> * at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:443)*
> * at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:630)*
> * at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:486)*
> * at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:483)*
> * at
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)*
> * at java.security.AccessController.doPrivileged(Native Method)*
> * at javax.security.auth.Subject.doAs(Subject.java:422)*
> * at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)*
> * at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)*
> * at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:483)*
> *Caused by: java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not
> found*
> * at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)*
> * at
> org.apache.hadoop.yarn.client.RMProxy.createRMFailoverProxyProvider(RMProxy.java:161)*
> * at
> org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:94)*
> * at
> org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)*
>
>
> I referred this mail thread also: http://apache-flink-
> user-mailing-list-archive.2336050.n4.nabble.com/Flink-session-on-Yarn-
> ClassNotFoundException-td15222.html
>
> I am using HDInsight Cluster(HDP3.6). According to the documentation,
> set HADOOP_CONF_DIR and YARN_CONF_DIR as well.
>
> Any inputs will be really helpful. Thanks!
>
> --
> Thanks & Regards,
> Nishu Tayal
>
--
Federico D'Ambrosio