You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Sumit Nigam <su...@yahoo.com.INVALID> on 2016/11/29 16:32:56 UTC
Multi-homed hosts with Hbase
Hi,
I am using multi-homed kerberized hosts with hbase 0.98.
The /etc/hosts files are uniformly containing FQDNs for cluster nodes. However, region server fails to register with master server with the error:
2016-11-17 15:05:47,700 WARN [regionserver54723] regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup connection for INFACSDEVKERB/dvterhpdw02.devadmin.nbsdev.co.uk@DEVADMIN.NBSDEV.CO.UK to INFACSDEVKERB/dvterhpdw03.devadmin.nbsdev.co.uk@DEVADMIN.NBSDEV.CO.UK
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1679)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1720)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2190)
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
The error indicates some DNS lookup problem. So, my question is do I need to enable some config in Hbase to allow multi-homed hosts? Also, I assume that if /etc/hosts is setup correctly, then entries in DNS server should not matter given the order of lookup is /etc/hosts and then DNS in my setup. I looked at some documentation and bugs (https://issues.apache.org/jira/browse/HBASE-12954 or https://issues.apache.org/jira/browse/HADOOP-8198) which seem to indicate multi-home support is not straight forward?
Zookeeper is not managed by Hbase. As a next step, I will dump zookeeper data from hbase shell to see what hostnames/ fqdns are being reported by master and region server. Any other suggestions can I try?
Thanks,
Sumit
Re: Multi-homed hosts with Hbase
Posted by Sumit Nigam <su...@yahoo.com.INVALID>.
Hi,
Any insights into this issue?
Thanks,Sumit
From: Sumit Nigam <su...@yahoo.com.INVALID>
To: Hbase-user <us...@hbase.apache.org>
Sent: Tuesday, November 29, 2016 10:02 PM
Subject: Multi-homed hosts with Hbase
Hi,
I am using multi-homed kerberized hosts with hbase 0.98.
The /etc/hosts files are uniformly containing FQDNs for cluster nodes. However, region server fails to register with master server with the error:
2016-11-17 15:05:47,700 WARN [regionserver54723] regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup connection for INFACSDEVKERB/abc02.devadmin.com@DEVADMIN.COM to INFACSDEVKERB/abc03.devadmin.com@DEVADMIN.COM
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1679)
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1720)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2190)
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
The error indicates some DNS lookup problem. So, my question is do I need to enable some config in Hbase to allow multi-homed hosts? Also, I assume that if /etc/hosts is setup correctly, then entries in DNS server should not matter given the order of lookup is /etc/hosts and then DNS in my setup. I looked at some documentation and bugs (https://issues.apache.org/jira/browse/HBASE-12954 or https://issues.apache.org/jira/browse/HADOOP-8198) which seem to indicate multi-home support is not straight forward?
Zookeeper is not managed by Hbase. As a next step, I will dump zookeeper data from hbase shell to see what hostnames/ fqdns are being reported by master and region server. Any other suggestions can I try?
Thanks,
Sumit