Kerberos Exceptions on Secured HBase Cluster

[陆恒 <lu...@xiaomi.com>]

Hi all:
       Recently I'm trying to run crunch jobs on a secured hbase cluster. However, as long as I have the admin access to the cluster in the shell mode, I cannot run the WordAggregationHBase in crunch-example due to exceptions below:

2013-11-06 12:07:21,628 ERROR [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:h_luheng (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2013-11-06 12:07:21,629 WARN [main] org.apache.hadoop.ipc.SecureClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2013-11-06 12:07:21,630 FATAL [main] org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
        at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:138)
        at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:177)
        at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:85)
        at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:286)
        at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:283)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        It seems that when configuring jobs, crunch does not do something about security setting, like "initCredentials(job)" as we can do when writing mr jobs on hbase.

        So did any one had seen problems like this? and how to fix it?
        Looking for your reply sincerely!

Best Regards
Ron