You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Jakub Stransky <st...@gmail.com> on 2016/01/28 10:19:50 UTC

Configuring hadoop in Azure on linux using Azure BLOB storage

Hello,

we are trying to configure hadoop HDP 2.2 running on azure cloud to use a
Azure Storage BLOB instead of regular HDFS.
Cluster is up and running, we can list files in azure blob storage over
hdoop fs commands. But when trying to run smoke test mapreduce teragen we
are getting following exception

16/01/28 09:10:11 INFO mapreduce.Cluster: Failed to use
org.apache.hadoop.mapred.YarnClientProtocolProvider due to error:
java.lang.reflect.InvocationTargetException
java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
        at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
        at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
        at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
        at org.apache.hadoop.examples.terasort.TeraGen.run(TeraGen.java:305)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at
org.apache.hadoop.examples.terasort.TeraGen.main(TeraGen.java:309)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at
org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Iam not sure what might be miss-configured. Can anybody help?

Thanks
Jakub