You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Laxman <la...@huawei.com> on 2012/05/22 11:51:29 UTC
Secure HBase setup
We got stuck with a problem while verifying client authentication in a secure HBase cluster.
We are able to start a secure HBase cluster successfully.
However, clients are not able to establish secure connection with HBase server successfully.
Other details:
HBase version: 0.94.0
Hadoop version: 0.23.1
Kerberos version: 1.10.1
Java version: 1.6.0_31, 64 bit
Linux version: SuSE 11.1 [Kernel version : 2.6.32.12-0.7-default x86_64 GNU/Linux]
We had gone thru the solutions available @
http://docs.oracle.com/javase/1.5.0/docs/guide/security/jgss/tutorials/Troubleshooting.html
https://ccp.cloudera.com/display/CDHDOC/Appendix+A+-+Troubleshooting#AppendixA-Troubleshooting-Problem2%3AJavaisunabletoreadtheKerberoscredentialscachecreatedbyversionsofMITKerberos1.8.1orhigher.
But none of then seems to work. Any clue?
There are no change in server logs as client is failing is failing even before it communicates with server.
Exception we are hitting (Client side logs):
2012-05-22 09:42:22,627 WARN org.apache.hadoop.ipc.SecureClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2012-05-22 09:42:22,627 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:testuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2012-05-22 09:42:22,630 DEBUG org.apache.hadoop.ipc.SecureClient: closing ipc connection to HOST-10-18-40-19/10.18.40.19:60020: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:227)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
at org.apache.hadoop.hbase.security.User.call(User.java:586)
at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440)
at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslConnectionFailure(SecureClient.java:194)
at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:274)
at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:485)
at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:69)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897)
at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:164)
at $Proxy6.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:208)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)
at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1284)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1240)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1227)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:936)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
at hbase.test.Hbasetest.main(Hbasetest.java:37)
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:138)
at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:176)
at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:84)
at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:267)
at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:264)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
at org.apache.hadoop.hbase.security.User.call(User.java:586)
at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440)
at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:263)
... 23 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
... 40 more
2012-05-22 09:42:22,636 DEBUG org.apache.hadoop.ipc.SecureClient: IPC Client (1778276127) connection to HOST-10-18-40-19/10.18.40.19:60020 from testuser: closed
2012-05-22 09:42:22,638 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=HOST-10-18-40-19, port=60020}, attempt=0 of 120 failed; retrying after sleep of 1000 because: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2012-05-22 09:42:22,640 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6ecf829d; serverName=HOST-10-18-40-19,60020,1337574445438
2012-05-22 09:42:23,641 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6ecf829d; serverName=HOST-10-18-40-19,60020,1337574445438
2012-05-22 09:42:23,642 DEBUG org.apache.hadoop.ipc.SecureClient: RPC Server Kerberos principal name for protocol=org.apache.hadoop.hbase.ipc.HRegionInterface is hbase/hadoop@HADOOP.COM
--
Regards,
Laxman
Re: Secure HBase setup
Posted by Yifeng Jiang <up...@gmail.com>.
Hi Laxman,
Have you obtained a Kerberos ticket before connecting to the cluster?
Can you try the following from your client and then reconnect to the cluster?
$ kinit testuser/your-client-hostname
-Yifeng
On May 22, 2012, at 6:51 PM, Laxman wrote:
> We got stuck with a problem while verifying client authentication in a secure HBase cluster.
> We are able to start a secure HBase cluster successfully.
>
> However, clients are not able to establish secure connection with HBase server successfully.
>
> Other details:
> HBase version: 0.94.0
> Hadoop version: 0.23.1
> Kerberos version: 1.10.1
> Java version: 1.6.0_31, 64 bit
> Linux version: SuSE 11.1 [Kernel version : 2.6.32.12-0.7-default x86_64 GNU/Linux]
>
> We had gone thru the solutions available @
> http://docs.oracle.com/javase/1.5.0/docs/guide/security/jgss/tutorials/Troubleshooting.html
> https://ccp.cloudera.com/display/CDHDOC/Appendix+A+-+Troubleshooting#AppendixA-Troubleshooting-Problem2%3AJavaisunabletoreadtheKerberoscredentialscachecreatedbyversionsofMITKerberos1.8.1orhigher.
>
> But none of then seems to work. Any clue?
>
> There are no change in server logs as client is failing is failing even before it communicates with server.
> Exception we are hitting (Client side logs):
>
> 2012-05-22 09:42:22,627 WARN org.apache.hadoop.ipc.SecureClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> 2012-05-22 09:42:22,627 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:testuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> 2012-05-22 09:42:22,630 DEBUG org.apache.hadoop.ipc.SecureClient: closing ipc connection to HOST-10-18-40-19/10.18.40.19:60020: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:227)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> at org.apache.hadoop.hbase.security.User.call(User.java:586)
> at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
> at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440)
> at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslConnectionFailure(SecureClient.java:194)
> at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:274)
> at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:485)
> at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:69)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897)
> at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:164)
> at $Proxy6.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:208)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1284)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1240)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1227)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:936)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
> at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> at hbase.test.Hbasetest.main(Hbasetest.java:37)
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
> at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:138)
> at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:176)
> at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:84)
> at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:267)
> at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:264)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> at org.apache.hadoop.hbase.security.User.call(User.java:586)
> at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
> at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440)
> at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:263)
> ... 23 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
> at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
> ... 40 more
> 2012-05-22 09:42:22,636 DEBUG org.apache.hadoop.ipc.SecureClient: IPC Client (1778276127) connection to HOST-10-18-40-19/10.18.40.19:60020 from testuser: closed
> 2012-05-22 09:42:22,638 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=HOST-10-18-40-19, port=60020}, attempt=0 of 120 failed; retrying after sleep of 1000 because: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> 2012-05-22 09:42:22,640 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6ecf829d; serverName=HOST-10-18-40-19,60020,1337574445438
> 2012-05-22 09:42:23,641 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6ecf829d; serverName=HOST-10-18-40-19,60020,1337574445438
> 2012-05-22 09:42:23,642 DEBUG org.apache.hadoop.ipc.SecureClient: RPC Server Kerberos principal name for protocol=org.apache.hadoop.hbase.ipc.HRegionInterface is hbase/hadoop@HADOOP.COM
>
>
> --
> Regards,
> Laxman
>
Re: Secure HBase setup
Posted by Andrew Purtell <ap...@apache.org>.
Thanks for taking the time to write back Laxman. I've proposed an
update to the troubleshooting section of the HBase manual, please see
https://issues.apache.org/jira/browse/HBASE-6077.
- Andy
On Tue, May 22, 2012 at 8:59 PM, Laxman <la...@huawei.com> wrote:
> This issue is resolved after replacing the Java JCE jars on client side as well.
> I feel its worth documenting in HBase book.
>
> --
> Regards,
> Laxman
>> -----Original Message-----
>> From: Laxman [mailto:lakshman.ch@huawei.com]
>> Sent: Tuesday, May 22, 2012 3:21 PM
>> To: dev@hbase.apache.org
>> Subject: Secure HBase setup
>>
>> We got stuck with a problem while verifying client authentication in a
>> secure HBase cluster.
>> We are able to start a secure HBase cluster successfully.
>>
>> However, clients are not able to establish secure connection with HBase
>> server successfully.
>>
>> Other details:
>> HBase version: 0.94.0
>> Hadoop version: 0.23.1
>> Kerberos version: 1.10.1
>> Java version: 1.6.0_31, 64 bit
>> Linux version: SuSE 11.1 [Kernel version : 2.6.32.12-0.7-default x86_64
>> GNU/Linux]
>>
>> We had gone thru the solutions available @
>> http://docs.oracle.com/javase/1.5.0/docs/guide/security/jgss/tutorials/
>> Troubleshooting.html
>> https://ccp.cloudera.com/display/CDHDOC/Appendix+A+-
>> +Troubleshooting#AppendixA-Troubleshooting-
>> Problem2%3AJavaisunabletoreadtheKerberoscredentialscachecreatedbyversio
>> nsofMITKerberos1.8.1orhigher.
>>
>> But none of then seems to work. Any clue?
>>
>> There are no change in server logs as client is failing is failing even
>> before it communicates with server.
>> Exception we are hitting (Client side logs):
>>
>> 2012-05-22 09:42:22,627 WARN org.apache.hadoop.ipc.SecureClient:
>> Exception encountered while connecting to the server :
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>> GSSException: No valid credentials provided (Mechanism level: Failed to
>> find any Kerberos tgt)]
>> 2012-05-22 09:42:22,627 ERROR
>> org.apache.hadoop.security.UserGroupInformation:
>> PriviledgedActionException as:testuser (auth:KERBEROS)
>> cause:java.io.IOException: javax.security.sasl.SaslException: GSS
>> initiate failed [Caused by GSSException: No valid credentials provided
>> (Mechanism level: Failed to find any Kerberos tgt)]
>> 2012-05-22 09:42:22,630 DEBUG org.apache.hadoop.ipc.SecureClient:
>> closing ipc connection to HOST-10-18-40-19/10.18.40.19:60020:
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>> GSSException: No valid credentials provided (Mechanism level: Failed to
>> find any Kerberos tgt)]
>> java.io.IOException: javax.security.sasl.SaslException: GSS initiate
>> failed [Caused by GSSException: No valid credentials provided
>> (Mechanism level: Failed to find any Kerberos tgt)]
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureC
>> lient.java:227)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
>> on.java:1177)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
>> va:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
>> rImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
>> at org.apache.hadoop.hbase.security.User.call(User.java:586)
>> at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
>> at
>> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:
>> 440)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslCon
>> nectionFailure(SecureClient.java:194)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstream
>> s(SecureClient.java:274)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.jav
>> a:485)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.jav
>> a:69)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897)
>> at
>> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEng
>> ine.java:164)
>> at $Proxy6.getProtocolVersion(Unknown Source)
>> at
>> org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.ja
>> va:208)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.getHRegionConnection(HConnectionManager.java:1284)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.getHRegionConnection(HConnectionManager.java:1240)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.getHRegionConnection(HConnectionManager.java:1227)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.locateRegionInMeta(HConnectionManager.java:936)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.locateRegion(HConnectionManager.java:832)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.locateRegion(HConnectionManager.java:801)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.locateRegionInMeta(HConnectionManager.java:933)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.locateRegion(HConnectionManager.java:836)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.locateRegion(HConnectionManager.java:801)
>> at
>> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
>> at hbase.test.Hbasetest.main(Hbasetest.java:37)
>> Caused by: javax.security.sasl.SaslException: GSS initiate failed
>> [Caused by GSSException: No valid credentials provided (Mechanism
>> level: Failed to find any Kerberos tgt)]
>> at
>> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Cl
>> ient.java:194)
>> at
>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSa
>> slRpcClient.java:138)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConn
>> ection(SecureClient.java:176)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(Se
>> cureClient.java:84)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureC
>> lient.java:267)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureC
>> lient.java:264)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
>> on.java:1177)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
>> va:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
>> rImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
>> at org.apache.hadoop.hbase.security.User.call(User.java:586)
>> at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
>> at
>> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:
>> 440)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstream
>> s(SecureClient.java:263)
>> ... 23 more
>> Caused by: GSSException: No valid credentials provided (Mechanism
>> level: Failed to find any Kerberos tgt)
>> at
>> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredentia
>> l.java:130)
>> at
>> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFac
>> tory.java:106)
>> at
>> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFact
>> ory.java:172)
>> at
>> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.jav
>> a:209)
>> at
>> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195
>> )
>> at
>> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162
>> )
>> at
>> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Cl
>> ient.java:175)
>> ... 40 more
>> 2012-05-22 09:42:22,636 DEBUG org.apache.hadoop.ipc.SecureClient: IPC
>> Client (1778276127) connection to HOST-10-18-40-19/10.18.40.19:60020
>> from testuser: closed
>> 2012-05-22 09:42:22,638 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-
>> ROOT-,,0.70236052, hostname=HOST-10-18-40-19, port=60020}, attempt=0 of
>> 120 failed; retrying after sleep of 1000 because:
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>> GSSException: No valid credentials provided (Mechanism level: Failed to
>> find any Kerberos tgt)]
>> 2012-05-22 09:42:22,640 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion: Looked up root region location,
>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectio
>> nImplementation@6ecf829d; serverName=HOST-10-18-40-
>> 19,60020,1337574445438
>> 2012-05-22 09:42:23,641 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion: Looked up root region location,
>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectio
>> nImplementation@6ecf829d; serverName=HOST-10-18-40-
>> 19,60020,1337574445438
>> 2012-05-22 09:42:23,642 DEBUG org.apache.hadoop.ipc.SecureClient: RPC
>> Server Kerberos principal name for
>> protocol=org.apache.hadoop.hbase.ipc.HRegionInterface is
>> hbase/hadoop@HADOOP.COM
>>
>>
>> --
>> Regards,
>> Laxman
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
RE: Secure HBase setup
Posted by Laxman <la...@huawei.com>.
This issue is resolved after replacing the Java JCE jars on client side as well.
I feel its worth documenting in HBase book.
--
Regards,
Laxman
> -----Original Message-----
> From: Laxman [mailto:lakshman.ch@huawei.com]
> Sent: Tuesday, May 22, 2012 3:21 PM
> To: dev@hbase.apache.org
> Subject: Secure HBase setup
>
> We got stuck with a problem while verifying client authentication in a
> secure HBase cluster.
> We are able to start a secure HBase cluster successfully.
>
> However, clients are not able to establish secure connection with HBase
> server successfully.
>
> Other details:
> HBase version: 0.94.0
> Hadoop version: 0.23.1
> Kerberos version: 1.10.1
> Java version: 1.6.0_31, 64 bit
> Linux version: SuSE 11.1 [Kernel version : 2.6.32.12-0.7-default x86_64
> GNU/Linux]
>
> We had gone thru the solutions available @
> http://docs.oracle.com/javase/1.5.0/docs/guide/security/jgss/tutorials/
> Troubleshooting.html
> https://ccp.cloudera.com/display/CDHDOC/Appendix+A+-
> +Troubleshooting#AppendixA-Troubleshooting-
> Problem2%3AJavaisunabletoreadtheKerberoscredentialscachecreatedbyversio
> nsofMITKerberos1.8.1orhigher.
>
> But none of then seems to work. Any clue?
>
> There are no change in server logs as client is failing is failing even
> before it communicates with server.
> Exception we are hitting (Client side logs):
>
> 2012-05-22 09:42:22,627 WARN org.apache.hadoop.ipc.SecureClient:
> Exception encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 2012-05-22 09:42:22,627 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:testuser (auth:KERBEROS)
> cause:java.io.IOException: javax.security.sasl.SaslException: GSS
> initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> 2012-05-22 09:42:22,630 DEBUG org.apache.hadoop.ipc.SecureClient:
> closing ipc connection to HOST-10-18-40-19/10.18.40.19:60020:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate
> failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureC
> lient.java:227)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
> on.java:1177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> va:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> rImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> at org.apache.hadoop.hbase.security.User.call(User.java:586)
> at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:
> 440)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslCon
> nectionFailure(SecureClient.java:194)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstream
> s(SecureClient.java:274)
> at
> org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.jav
> a:485)
> at
> org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.jav
> a:69)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897)
> at
> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEng
> ine.java:164)
> at $Proxy6.getProtocolVersion(Unknown Source)
> at
> org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.ja
> va:208)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
> tion.getHRegionConnection(HConnectionManager.java:1284)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
> tion.getHRegionConnection(HConnectionManager.java:1240)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
> tion.getHRegionConnection(HConnectionManager.java:1227)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
> tion.locateRegionInMeta(HConnectionManager.java:936)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
> tion.locateRegion(HConnectionManager.java:832)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
> tion.locateRegion(HConnectionManager.java:801)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
> tion.locateRegionInMeta(HConnectionManager.java:933)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
> tion.locateRegion(HConnectionManager.java:836)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
> tion.locateRegion(HConnectionManager.java:801)
> at
> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> at hbase.test.Hbasetest.main(Hbasetest.java:37)
> Caused by: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism
> level: Failed to find any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Cl
> ient.java:194)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSa
> slRpcClient.java:138)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConn
> ection(SecureClient.java:176)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(Se
> cureClient.java:84)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureC
> lient.java:267)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureC
> lient.java:264)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
> on.java:1177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
> va:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
> rImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> at org.apache.hadoop.hbase.security.User.call(User.java:586)
> at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:
> 440)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstream
> s(SecureClient.java:263)
> ... 23 more
> Caused by: GSSException: No valid credentials provided (Mechanism
> level: Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredentia
> l.java:130)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFac
> tory.java:106)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFact
> ory.java:172)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.jav
> a:209)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195
> )
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162
> )
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Cl
> ient.java:175)
> ... 40 more
> 2012-05-22 09:42:22,636 DEBUG org.apache.hadoop.ipc.SecureClient: IPC
> Client (1778276127) connection to HOST-10-18-40-19/10.18.40.19:60020
> from testuser: closed
> 2012-05-22 09:42:22,638 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
> tion: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-
> ROOT-,,0.70236052, hostname=HOST-10-18-40-19, port=60020}, attempt=0 of
> 120 failed; retrying after sleep of 1000 because:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 2012-05-22 09:42:22,640 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
> tion: Looked up root region location,
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectio
> nImplementation@6ecf829d; serverName=HOST-10-18-40-
> 19,60020,1337574445438
> 2012-05-22 09:42:23,641 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
> tion: Looked up root region location,
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectio
> nImplementation@6ecf829d; serverName=HOST-10-18-40-
> 19,60020,1337574445438
> 2012-05-22 09:42:23,642 DEBUG org.apache.hadoop.ipc.SecureClient: RPC
> Server Kerberos principal name for
> protocol=org.apache.hadoop.hbase.ipc.HRegionInterface is
> hbase/hadoop@HADOOP.COM
>
>
> --
> Regards,
> Laxman