Unable to access HBase from a Mapper

[Barrie Kersbergen <bk...@bol.com>]

Hi Subroto,

I had exactly the same issue (although 1.5 years after your question) and I couldn't find a working solution in the documentation or on the internet.
It took me a while to figure this out so I put my working example on github.

https://github.com/bkersbergen/hbase-kerberos-mapreduce-hadoop2

Kind regards,
Barrie


Hi,

I have an MapReduce job which uses does some operation of reading from HBase tables. I have
configured the cluster in Secure Mode including Secure HBase.

I am running the Job(classical MR job) from a custom client running under user "subroto".
The mentioned user has valid principal in kerberos but, when the Job runs it gets the exceptions:
2012-11-14 10:55:24,486 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:subroto (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused
by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)]
2012-11-14 10:55:24,490 WARN org.apache.hadoop.ipc.SecureClient: Exception encountered while
connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)]
2012-11-14 10:55:24,493 FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed.
The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials
provided (Mechanism level: Failed to find any Kerberos tgt)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
        at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:138)
        at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:177)
        at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:85)
        at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:284)
        at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:281)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
        at org.apache.hadoop.hbase.security.User.call(User.java:586)
        at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
        at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440)
        at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:280)
        at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:502)
        at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:70)
        at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897)
        at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:164)
        at $Proxy13.getProtocolVersion(Unknown Source)
        at org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:208)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)
        at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:642)
        at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:108)

I can access the tables from same user via HBase shell and the same job is running fine in
Local Job Runner mode.
Other jobs which don't access HBase work perfectly fine in the cluster.

I am using Hadoop Version: 2.0.0
The Hbase version being used is HBase 0.92.1

Please let me know if some thing I need to check or I am missing.

Cheers,
Subroto Sanyal