Unable to access HBase from a Mapper

[Subroto <ss...@datameer.com>]

Hi,

I have an MapReduce job which uses does some operation of reading from HBase tables. I have configured the cluster in Secure Mode including Secure HBase.

I am running the Job(classical MR job) from a custom client running under user "subroto".
The mentioned user has valid principal in kerberos but, when the Job runs it gets the exceptions:
2012-11-14 10:55:24,486 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:subroto (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2012-11-14 10:55:24,490 WARN org.apache.hadoop.ipc.SecureClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2012-11-14 10:55:24,493 FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
	at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:138)
	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:177)
	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:85)
	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:284)
	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:281)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
	at org.apache.hadoop.hbase.security.User.call(User.java:586)
	at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440)
	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:280)
	at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:502)
	at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:70)
	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897)
	at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:164)
	at $Proxy13.getProtocolVersion(Unknown Source)
	at org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:208)
	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303)
	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)
	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:642)
	at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:108)

I can access the tables from same user via HBase shell and the same job is running fine in Local Job Runner mode.
Other jobs which don't access HBase work perfectly fine in the cluster.

I am using Hadoop Version: 2.0.0
The Hbase version being used is HBase 0.92.1

Please let me know if some thing I need to check or I am missing.

Cheers,
Subroto Sanyal