Error while setting up flink on HDInsight cluster

[Nishu <ni...@gmail.com>]

Hi,

I am trying to run flink on top on HDInsight cluster.  So far I have added
all libraries in classpath and set YARN_CONF_DIR, HADOOP_CONF_DIR and
HADOOP_CLASSPATH. It's running on hadoop 2.7.3.

When I run yarn-session.sh, It throws following error.


*2017-11-21 13:07:01,946 INFO
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl     - Timeline
service address: http://headnodehost:8188/ws/v1/timeline/
<http://headnodehost:8188/ws/v1/timeline/>*
*Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V*
*        at
org.apache.hadoop.fs.adl.AdlConfKeys.addDeprecatedKeys(AdlConfKeys.java:114)*
*        at
org.apache.hadoop.fs.adl.AdlFileSystem.<clinit>(AdlFileSystem.java:92)*
*        at java.lang.Class.forName0(Native Method)*
*        at java.lang.Class.forName(Class.java:348)*
*        at
org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)*
*        at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)*
*        at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)*
*        at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)*
*        at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)*
*        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)*
*        at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)*
*        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)*
*        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)*
*        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)*
*        at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.startAppMaster(AbstractYarnClusterDescriptor.java:609)*
*        at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:582)*
*        at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:441)*
*        at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:630)*
*        at
org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:486)*
*        at
org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:483)*
*        at
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)*
*        at java.security.AccessController.doPrivileged(Native Method)*
*        at javax.security.auth.Subject.doAs(Subject.java:422)*
*        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)*
*        at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)*

I decompiled the hadoop jar to see, if reloadExistingConfiguration method
is present in Configuration class. That's available too.

Has anyone faced the same before? Any help is really appreciated. Thanks in
advance.

-- 
Thanks & Regards,
Nishu Tayal