You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Loïc Chanel <lo...@telecomnancy.net> on 2015/08/20 14:46:45 UTC
Problem with HBase + Kerberos
Hi all,
Since I kerberized my cluster, it seems like I can't use HBase anymore ...
For example, executing create 'toto','titi' on HBase shell results in the
printing of this line endlessly :
WARN [main] security.UserGroupInformation: Not attempting to re-login
since the last re-login was attempted less than 600 seconds before.
And nothing else happens.
I tried to restart HDFS and HBase, and to re-generate credentials and
keytabs, but nothing changed.
As for the logs, they are not very explicits, as the only thing they say
(and keep saying) is :
2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
ipc.RpcServer: Created SASL server with mechanism = GSSAPI
2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
ipc.RpcServer: Have read input token of size 650 for processing by
saslServer.evaluateResponse()
2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
ipc.RpcServer: Will send token of size 108 from saslServer.
2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
ipc.RpcServer: Have read input token of size 0 for processing by
saslServer.evaluateResponse()
2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
ipc.RpcServer: Will send token of size 32 from saslServer.
2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING client
192.168.6.148:43014 because read count=-1. Number of active connections: 3
Do anyone has an idea about where this might come from, or how to solve it
? Because I couldn't find much documentation about this.
Thanks in advance for your help !
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
Re: Problem with HBase + Kerberos
Posted by Anil Gupta <an...@gmail.com>.
Yes there is expiration of kerberos session. But, it renews by itself.
As per log, your problem seems to be related to invalid credentials or trouble doing login.
> Couldn't
> setup connection for hbase/host@REALM.WL to hbase/server@REALM.WL
Sent from my iPhone
> On Aug 31, 2015, at 1:43 AM, Loïc Chanel <lo...@telecomnancy.net> wrote:
>
> Actually, it seems like it is related to Kerberos, as I keep having these
> lines in RegionServers' logs :
>
> 2015-08-31 10:15:23,370 DEBUG [regionserver60020]
> security.HBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos
> principal name is hbase/server@REALM.WL
> 2015-08-31 10:15:23,372 DEBUG [regionserver60020] ipc.RpcClient: Exception
> encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> 2015-08-31 10:15:23,372 WARN [regionserver60020]
> security.UserGroupInformation: Not attempting to re-login since the last
> re-login was attempted less than 600 seconds before.
> 2015-08-31 10:15:27,908 DEBUG [regionserver60020]
> security.HBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos
> principal name is hbase/server@REALM.WL
> 2015-08-31 10:15:27,910 WARN [regionserver60020] ipc.RpcClient:
> 2015-08-31 10:15:27,910 WARN [regionserver60020]
> regionserver.HRegionServer: error telling master we are up
> com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup
> connection forhbase/host@REALM.WL to hbase/server@REALM.WL
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1739)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
> at
> org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:5402)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2114)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:877)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Couldn't setup connection for
> hbase/host@REALM.WL to hbase/server@REALM.WL
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
> ... 5 more
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(Unknown
> Source)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:177)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
> ... 9 more
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Unknown
> Source)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Unknown Source)
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(Unknown
> Source)
> at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
> at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
> ... 19 more
> 2015-08-31 10:15:27,911 WARN [regionserver60020]
> regionserver.HRegionServer: reportForDuty failed; sleeping and then
> retrying.
>
> Is there kind of an expiration limit for keytab credentials ?
> Thanks for your help,
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-08-27 18:24 GMT+02:00 anil gupta <an...@gmail.com>:
>
>> Maybe, this is related to some Ambari setup? Can you also ask on Ambari
>> mailing list.
>> IMO, secure HBase cluster connectivity has been working in HBase for a very
>> long time.
>>
>> On Thu, Aug 27, 2015 at 12:48 AM, Loïc Chanel <
>> loic.chanel@telecomnancy.net>
>> wrote:
>>
>>> I did not, but as I Kerberized my cluster with Ambari, it did the
>> mandatory
>>> modifications.
>>>
>>> Loïc CHANEL
>>> Engineering student at TELECOM Nancy
>>> Trainee at Worldline - Villeurbanne
>>>
>>> 2015-08-27 1:17 GMT+02:00 Laurent H <la...@gmail.com>:
>>>
>>>> Do you change some stuff in your hbase-site.xml when you've installed
>>>> Kerberos ?
>>>>
>>>> --
>>>> Laurent HATIER - Consultant Big Data & Business Intelligence chez
>>> CapGemini
>>>> fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
>>>> <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
>>>>
>>>> 2015-08-21 9:44 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>>>>
>>>>> Sorry if I didn't mention that, but yeah, I ran kinit before invoking
>>>> hbase
>>>>> shell, and klists command says that my user has a ticket.
>>>>> [root@host /]# klist
>>>>> Ticket cache: FILE:/tmp/krb5cc_0
>>>>> Default principal: testuser@REALM
>>>>>
>>>>> Valid starting Expires Service principal
>>>>> 08/21/15 09:39:33 08/22/15 09:39:33 krbtgt/REALM@REALM
>>>>> renew until 08/21/15 09:39:33
>>>>>
>>>>>
>>>>> Loïc CHANEL
>>>>> Engineering student at TELECOM Nancy
>>>>> Trainee at Worldline - Villeurbanne
>>>>>
>>>>> 2015-08-21 6:12 GMT+02:00 anil gupta <an...@gmail.com>:
>>>>>
>>>>>> Did you run kinit command before invoking "hbase shell"? What does
>>>> klist
>>>>>> command says?
>>>>>>
>>>>>> On Thu, Aug 20, 2015 at 6:47 AM, Loïc Chanel <
>>>>> loic.chanel@telecomnancy.net
>>>>>> wrote:
>>>>>>
>>>>>>> By the way, as this may help to find my issue, I just tested
>> typing
>>>>>> *whoami
>>>>>>> *in HBase shell : this returned me exactly what it should :
>>>>>>> testuser@REALM (auth:KERBEROS)
>>>>>>> groups: nobody, toast
>>>>>>>
>>>>>>> Loïc CHANEL
>>>>>>> Engineering student at TELECOM Nancy
>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>
>>>>>>> 2015-08-20 15:17 GMT+02:00 Loïc Chanel <
>>> loic.chanel@telecomnancy.net
>>>>> :
>>>>>>>
>>>>>>>> Nothing more with your option :/
>>>>>>>>
>>>>>>>> Loïc CHANEL
>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>>
>>>>>>>> 2015-08-20 15:04 GMT+02:00 Loïc Chanel <
>>>> loic.chanel@telecomnancy.net
>>>>>> :
>>>>>>>>
>>>>>>>>> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
>>>>>>>>> I have unlimited strength JCE installed.
>>>>>>>>>
>>>>>>>>> I'll try to have more clues with this option.
>>>>>>>>>
>>>>>>>>> Loïc CHANEL
>>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>>>>
>>>>>>>>> 2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
>>>>>>>>>
>>>>>>>>>> Which hbase / hadoop release are you using ?
>>>>>>>>>>
>>>>>>>>>> Running with -Dsun.security.krb5.debug=true will provide more
>>>> clue.
>>>>>>>>>>
>>>>>>>>>> Do you have unlimited strength JCE installed ?
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
>>>>>>>>>> loic.chanel@telecomnancy.net>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> Since I kerberized my cluster, it seems like I can't use
>>> HBase
>>>>>>> anymore
>>>>>>>>>> ...
>>>>>>>>>>> For example, executing create 'toto','titi' on HBase shell
>>>>> results
>>>>>>> in
>>>>>>>>>> the
>>>>>>>>>>> printing of this line endlessly :
>>>>>>>>>>> WARN [main] security.UserGroupInformation: Not attempting
>> to
>>>>>>> re-login
>>>>>>>>>>> since the last re-login was attempted less than 600 seconds
>>>>> before.
>>>>>>>>>>>
>>>>>>>>>>> And nothing else happens.
>>>>>>>>>>> I tried to restart HDFS and HBase, and to re-generate
>>>> credentials
>>>>>> and
>>>>>>>>>>> keytabs, but nothing changed.
>>>>>>>>>>> As for the logs, they are not very explicits, as the only
>>> thing
>>>>>> they
>>>>>>>>>> say
>>>>>>>>>>> (and keep saying) is :
>>>>>>>>>>>
>>>>>>>>>>> 2015-08-20 13:50:12,697 DEBUG
>> [RpcServer.reader=2,port=60000]
>>>>>>>>>>> ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>>>>>>>>>>> 2015-08-20 13:50:12,698 DEBUG
>> [RpcServer.reader=2,port=60000]
>>>>>>>>>>> ipc.RpcServer: Have read input token of size 650 for
>>> processing
>>>>> by
>>>>>>>>>>> saslServer.evaluateResponse()
>>>>>>>>>>> 2015-08-20 13:50:12,704 DEBUG
>> [RpcServer.reader=2,port=60000]
>>>>>>>>>>> ipc.RpcServer: Will send token of size 108 from saslServer.
>>>>>>>>>>> 2015-08-20 13:50:12,706 DEBUG
>> [RpcServer.reader=2,port=60000]
>>>>>>>>>>> ipc.RpcServer: Have read input token of size 0 for
>> processing
>>>> by
>>>>>>>>>>> saslServer.evaluateResponse()
>>>>>>>>>>> 2015-08-20 13:50:12,707 DEBUG
>> [RpcServer.reader=2,port=60000]
>>>>>>>>>>> ipc.RpcServer: Will send token of size 32 from saslServer.
>>>>>>>>>>> 2015-08-20 13:50:12,708 DEBUG
>> [RpcServer.reader=2,port=60000]
>>>>>>>>>>> ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING
>>>>> client
>>>>>>>>>>> 192.168.6.148:43014 because read count=-1. Number of
>> active
>>>>>>>>>> connections: 3
>>>>>>>>>>>
>>>>>>>>>>> Do anyone has an idea about where this might come from, or
>>> how
>>>> to
>>>>>>>>>> solve it
>>>>>>>>>>> ? Because I couldn't find much documentation about this.
>>>>>>>>>>> Thanks in advance for your help !
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Loïc
>>>>>>>>>>>
>>>>>>>>>>> Loïc CHANEL
>>>>>>>>>>> Engineering student at TELECOM Nancy
>>>>>>>>>>> Trainee at Worldline - Villeurbanne
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thanks & Regards,
>>>>>> Anil Gupta
>>
>>
>>
>> --
>> Thanks & Regards,
>> Anil Gupta
>>
Re: Problem with HBase + Kerberos
Posted by Loïc Chanel <lo...@telecomnancy.net>.
Actually, it seems like it is related to Kerberos, as I keep having these
lines in RegionServers' logs :
2015-08-31 10:15:23,370 DEBUG [regionserver60020]
security.HBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos
principal name is hbase/server@REALM.WL
2015-08-31 10:15:23,372 DEBUG [regionserver60020] ipc.RpcClient: Exception
encountered while connecting to the server :
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]
2015-08-31 10:15:23,372 WARN [regionserver60020]
security.UserGroupInformation: Not attempting to re-login since the last
re-login was attempted less than 600 seconds before.
2015-08-31 10:15:27,908 DEBUG [regionserver60020]
security.HBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos
principal name is hbase/server@REALM.WL
2015-08-31 10:15:27,910 WARN [regionserver60020] ipc.RpcClient: Couldn't
setup connection for hbase/host@REALM.WL to hbase/server@REALM.WL
2015-08-31 10:15:27,910 WARN [regionserver60020]
regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup
connection forhbase/host@REALM.WL to hbase/server@REALM.WL
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1739)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
at
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:5402)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2114)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:877)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Couldn't setup connection for
hbase/host@REALM.WL to hbase/server@REALM.WL
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
... 5 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused
by GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(Unknown
Source)
at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:177)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
... 9 more
Caused by: GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Unknown
Source)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
at
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Unknown Source)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(Unknown
Source)
at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
... 19 more
2015-08-31 10:15:27,911 WARN [regionserver60020]
regionserver.HRegionServer: reportForDuty failed; sleeping and then
retrying.
Is there kind of an expiration limit for keytab credentials ?
Thanks for your help,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-08-27 18:24 GMT+02:00 anil gupta <an...@gmail.com>:
> Maybe, this is related to some Ambari setup? Can you also ask on Ambari
> mailing list.
> IMO, secure HBase cluster connectivity has been working in HBase for a very
> long time.
>
> On Thu, Aug 27, 2015 at 12:48 AM, Loïc Chanel <
> loic.chanel@telecomnancy.net>
> wrote:
>
> > I did not, but as I Kerberized my cluster with Ambari, it did the
> mandatory
> > modifications.
> >
> > Loïc CHANEL
> > Engineering student at TELECOM Nancy
> > Trainee at Worldline - Villeurbanne
> >
> > 2015-08-27 1:17 GMT+02:00 Laurent H <la...@gmail.com>:
> >
> > > Do you change some stuff in your hbase-site.xml when you've installed
> > > Kerberos ?
> > >
> > > --
> > > Laurent HATIER - Consultant Big Data & Business Intelligence chez
> > CapGemini
> > > fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
> > > <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
> > >
> > > 2015-08-21 9:44 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> > >
> > > > Sorry if I didn't mention that, but yeah, I ran kinit before invoking
> > > hbase
> > > > shell, and klists command says that my user has a ticket.
> > > > [root@host /]# klist
> > > > Ticket cache: FILE:/tmp/krb5cc_0
> > > > Default principal: testuser@REALM
> > > >
> > > > Valid starting Expires Service principal
> > > > 08/21/15 09:39:33 08/22/15 09:39:33 krbtgt/REALM@REALM
> > > > renew until 08/21/15 09:39:33
> > > >
> > > >
> > > > Loïc CHANEL
> > > > Engineering student at TELECOM Nancy
> > > > Trainee at Worldline - Villeurbanne
> > > >
> > > > 2015-08-21 6:12 GMT+02:00 anil gupta <an...@gmail.com>:
> > > >
> > > > > Did you run kinit command before invoking "hbase shell"? What does
> > > klist
> > > > > command says?
> > > > >
> > > > > On Thu, Aug 20, 2015 at 6:47 AM, Loïc Chanel <
> > > > loic.chanel@telecomnancy.net
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > By the way, as this may help to find my issue, I just tested
> typing
> > > > > *whoami
> > > > > > *in HBase shell : this returned me exactly what it should :
> > > > > > testuser@REALM (auth:KERBEROS)
> > > > > > groups: nobody, toast
> > > > > >
> > > > > > Loïc CHANEL
> > > > > > Engineering student at TELECOM Nancy
> > > > > > Trainee at Worldline - Villeurbanne
> > > > > >
> > > > > > 2015-08-20 15:17 GMT+02:00 Loïc Chanel <
> > loic.chanel@telecomnancy.net
> > > >:
> > > > > >
> > > > > > > Nothing more with your option :/
> > > > > > >
> > > > > > > Loïc CHANEL
> > > > > > > Engineering student at TELECOM Nancy
> > > > > > > Trainee at Worldline - Villeurbanne
> > > > > > >
> > > > > > > 2015-08-20 15:04 GMT+02:00 Loïc Chanel <
> > > loic.chanel@telecomnancy.net
> > > > >:
> > > > > > >
> > > > > > >> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
> > > > > > >> I have unlimited strength JCE installed.
> > > > > > >>
> > > > > > >> I'll try to have more clues with this option.
> > > > > > >>
> > > > > > >> Loïc CHANEL
> > > > > > >> Engineering student at TELECOM Nancy
> > > > > > >> Trainee at Worldline - Villeurbanne
> > > > > > >>
> > > > > > >> 2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
> > > > > > >>
> > > > > > >>> Which hbase / hadoop release are you using ?
> > > > > > >>>
> > > > > > >>> Running with -Dsun.security.krb5.debug=true will provide more
> > > clue.
> > > > > > >>>
> > > > > > >>> Do you have unlimited strength JCE installed ?
> > > > > > >>>
> > > > > > >>> Cheers
> > > > > > >>>
> > > > > > >>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
> > > > > > >>> loic.chanel@telecomnancy.net>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>> > Hi all,
> > > > > > >>> >
> > > > > > >>> > Since I kerberized my cluster, it seems like I can't use
> > HBase
> > > > > > anymore
> > > > > > >>> ...
> > > > > > >>> > For example, executing create 'toto','titi' on HBase shell
> > > > results
> > > > > > in
> > > > > > >>> the
> > > > > > >>> > printing of this line endlessly :
> > > > > > >>> > WARN [main] security.UserGroupInformation: Not attempting
> to
> > > > > > re-login
> > > > > > >>> > since the last re-login was attempted less than 600 seconds
> > > > before.
> > > > > > >>> >
> > > > > > >>> > And nothing else happens.
> > > > > > >>> > I tried to restart HDFS and HBase, and to re-generate
> > > credentials
> > > > > and
> > > > > > >>> > keytabs, but nothing changed.
> > > > > > >>> > As for the logs, they are not very explicits, as the only
> > thing
> > > > > they
> > > > > > >>> say
> > > > > > >>> > (and keep saying) is :
> > > > > > >>> >
> > > > > > >>> > 2015-08-20 13:50:12,697 DEBUG
> [RpcServer.reader=2,port=60000]
> > > > > > >>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> > > > > > >>> > 2015-08-20 13:50:12,698 DEBUG
> [RpcServer.reader=2,port=60000]
> > > > > > >>> > ipc.RpcServer: Have read input token of size 650 for
> > processing
> > > > by
> > > > > > >>> > saslServer.evaluateResponse()
> > > > > > >>> > 2015-08-20 13:50:12,704 DEBUG
> [RpcServer.reader=2,port=60000]
> > > > > > >>> > ipc.RpcServer: Will send token of size 108 from saslServer.
> > > > > > >>> > 2015-08-20 13:50:12,706 DEBUG
> [RpcServer.reader=2,port=60000]
> > > > > > >>> > ipc.RpcServer: Have read input token of size 0 for
> processing
> > > by
> > > > > > >>> > saslServer.evaluateResponse()
> > > > > > >>> > 2015-08-20 13:50:12,707 DEBUG
> [RpcServer.reader=2,port=60000]
> > > > > > >>> > ipc.RpcServer: Will send token of size 32 from saslServer.
> > > > > > >>> > 2015-08-20 13:50:12,708 DEBUG
> [RpcServer.reader=2,port=60000]
> > > > > > >>> > ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING
> > > > client
> > > > > > >>> > 192.168.6.148:43014 because read count=-1. Number of
> active
> > > > > > >>> connections: 3
> > > > > > >>> >
> > > > > > >>> > Do anyone has an idea about where this might come from, or
> > how
> > > to
> > > > > > >>> solve it
> > > > > > >>> > ? Because I couldn't find much documentation about this.
> > > > > > >>> > Thanks in advance for your help !
> > > > > > >>> >
> > > > > > >>> >
> > > > > > >>> > Loïc
> > > > > > >>> >
> > > > > > >>> > Loïc CHANEL
> > > > > > >>> > Engineering student at TELECOM Nancy
> > > > > > >>> > Trainee at Worldline - Villeurbanne
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks & Regards,
> > > > > Anil Gupta
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
Re: Problem with HBase + Kerberos
Posted by anil gupta <an...@gmail.com>.
Maybe, this is related to some Ambari setup? Can you also ask on Ambari
mailing list.
IMO, secure HBase cluster connectivity has been working in HBase for a very
long time.
On Thu, Aug 27, 2015 at 12:48 AM, Loïc Chanel <lo...@telecomnancy.net>
wrote:
> I did not, but as I Kerberized my cluster with Ambari, it did the mandatory
> modifications.
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-08-27 1:17 GMT+02:00 Laurent H <la...@gmail.com>:
>
> > Do you change some stuff in your hbase-site.xml when you've installed
> > Kerberos ?
> >
> > --
> > Laurent HATIER - Consultant Big Data & Business Intelligence chez
> CapGemini
> > fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
> > <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
> >
> > 2015-08-21 9:44 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> >
> > > Sorry if I didn't mention that, but yeah, I ran kinit before invoking
> > hbase
> > > shell, and klists command says that my user has a ticket.
> > > [root@host /]# klist
> > > Ticket cache: FILE:/tmp/krb5cc_0
> > > Default principal: testuser@REALM
> > >
> > > Valid starting Expires Service principal
> > > 08/21/15 09:39:33 08/22/15 09:39:33 krbtgt/REALM@REALM
> > > renew until 08/21/15 09:39:33
> > >
> > >
> > > Loïc CHANEL
> > > Engineering student at TELECOM Nancy
> > > Trainee at Worldline - Villeurbanne
> > >
> > > 2015-08-21 6:12 GMT+02:00 anil gupta <an...@gmail.com>:
> > >
> > > > Did you run kinit command before invoking "hbase shell"? What does
> > klist
> > > > command says?
> > > >
> > > > On Thu, Aug 20, 2015 at 6:47 AM, Loïc Chanel <
> > > loic.chanel@telecomnancy.net
> > > > >
> > > > wrote:
> > > >
> > > > > By the way, as this may help to find my issue, I just tested typing
> > > > *whoami
> > > > > *in HBase shell : this returned me exactly what it should :
> > > > > testuser@REALM (auth:KERBEROS)
> > > > > groups: nobody, toast
> > > > >
> > > > > Loïc CHANEL
> > > > > Engineering student at TELECOM Nancy
> > > > > Trainee at Worldline - Villeurbanne
> > > > >
> > > > > 2015-08-20 15:17 GMT+02:00 Loïc Chanel <
> loic.chanel@telecomnancy.net
> > >:
> > > > >
> > > > > > Nothing more with your option :/
> > > > > >
> > > > > > Loïc CHANEL
> > > > > > Engineering student at TELECOM Nancy
> > > > > > Trainee at Worldline - Villeurbanne
> > > > > >
> > > > > > 2015-08-20 15:04 GMT+02:00 Loïc Chanel <
> > loic.chanel@telecomnancy.net
> > > >:
> > > > > >
> > > > > >> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
> > > > > >> I have unlimited strength JCE installed.
> > > > > >>
> > > > > >> I'll try to have more clues with this option.
> > > > > >>
> > > > > >> Loïc CHANEL
> > > > > >> Engineering student at TELECOM Nancy
> > > > > >> Trainee at Worldline - Villeurbanne
> > > > > >>
> > > > > >> 2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
> > > > > >>
> > > > > >>> Which hbase / hadoop release are you using ?
> > > > > >>>
> > > > > >>> Running with -Dsun.security.krb5.debug=true will provide more
> > clue.
> > > > > >>>
> > > > > >>> Do you have unlimited strength JCE installed ?
> > > > > >>>
> > > > > >>> Cheers
> > > > > >>>
> > > > > >>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
> > > > > >>> loic.chanel@telecomnancy.net>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> > Hi all,
> > > > > >>> >
> > > > > >>> > Since I kerberized my cluster, it seems like I can't use
> HBase
> > > > > anymore
> > > > > >>> ...
> > > > > >>> > For example, executing create 'toto','titi' on HBase shell
> > > results
> > > > > in
> > > > > >>> the
> > > > > >>> > printing of this line endlessly :
> > > > > >>> > WARN [main] security.UserGroupInformation: Not attempting to
> > > > > re-login
> > > > > >>> > since the last re-login was attempted less than 600 seconds
> > > before.
> > > > > >>> >
> > > > > >>> > And nothing else happens.
> > > > > >>> > I tried to restart HDFS and HBase, and to re-generate
> > credentials
> > > > and
> > > > > >>> > keytabs, but nothing changed.
> > > > > >>> > As for the logs, they are not very explicits, as the only
> thing
> > > > they
> > > > > >>> say
> > > > > >>> > (and keep saying) is :
> > > > > >>> >
> > > > > >>> > 2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
> > > > > >>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> > > > > >>> > 2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
> > > > > >>> > ipc.RpcServer: Have read input token of size 650 for
> processing
> > > by
> > > > > >>> > saslServer.evaluateResponse()
> > > > > >>> > 2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
> > > > > >>> > ipc.RpcServer: Will send token of size 108 from saslServer.
> > > > > >>> > 2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
> > > > > >>> > ipc.RpcServer: Have read input token of size 0 for processing
> > by
> > > > > >>> > saslServer.evaluateResponse()
> > > > > >>> > 2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
> > > > > >>> > ipc.RpcServer: Will send token of size 32 from saslServer.
> > > > > >>> > 2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
> > > > > >>> > ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING
> > > client
> > > > > >>> > 192.168.6.148:43014 because read count=-1. Number of active
> > > > > >>> connections: 3
> > > > > >>> >
> > > > > >>> > Do anyone has an idea about where this might come from, or
> how
> > to
> > > > > >>> solve it
> > > > > >>> > ? Because I couldn't find much documentation about this.
> > > > > >>> > Thanks in advance for your help !
> > > > > >>> >
> > > > > >>> >
> > > > > >>> > Loïc
> > > > > >>> >
> > > > > >>> > Loïc CHANEL
> > > > > >>> > Engineering student at TELECOM Nancy
> > > > > >>> > Trainee at Worldline - Villeurbanne
> > > > > >>> >
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks & Regards,
> > > > Anil Gupta
> > > >
> > >
> >
>
--
Thanks & Regards,
Anil Gupta
Re: Problem with HBase + Kerberos
Posted by Loïc Chanel <lo...@telecomnancy.net>.
I did not, but as I Kerberized my cluster with Ambari, it did the mandatory
modifications.
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-08-27 1:17 GMT+02:00 Laurent H <la...@gmail.com>:
> Do you change some stuff in your hbase-site.xml when you've installed
> Kerberos ?
>
> --
> Laurent HATIER - Consultant Big Data & Business Intelligence chez CapGemini
> fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
> <http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
>
> 2015-08-21 9:44 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>
> > Sorry if I didn't mention that, but yeah, I ran kinit before invoking
> hbase
> > shell, and klists command says that my user has a ticket.
> > [root@host /]# klist
> > Ticket cache: FILE:/tmp/krb5cc_0
> > Default principal: testuser@REALM
> >
> > Valid starting Expires Service principal
> > 08/21/15 09:39:33 08/22/15 09:39:33 krbtgt/REALM@REALM
> > renew until 08/21/15 09:39:33
> >
> >
> > Loïc CHANEL
> > Engineering student at TELECOM Nancy
> > Trainee at Worldline - Villeurbanne
> >
> > 2015-08-21 6:12 GMT+02:00 anil gupta <an...@gmail.com>:
> >
> > > Did you run kinit command before invoking "hbase shell"? What does
> klist
> > > command says?
> > >
> > > On Thu, Aug 20, 2015 at 6:47 AM, Loïc Chanel <
> > loic.chanel@telecomnancy.net
> > > >
> > > wrote:
> > >
> > > > By the way, as this may help to find my issue, I just tested typing
> > > *whoami
> > > > *in HBase shell : this returned me exactly what it should :
> > > > testuser@REALM (auth:KERBEROS)
> > > > groups: nobody, toast
> > > >
> > > > Loïc CHANEL
> > > > Engineering student at TELECOM Nancy
> > > > Trainee at Worldline - Villeurbanne
> > > >
> > > > 2015-08-20 15:17 GMT+02:00 Loïc Chanel <loic.chanel@telecomnancy.net
> >:
> > > >
> > > > > Nothing more with your option :/
> > > > >
> > > > > Loïc CHANEL
> > > > > Engineering student at TELECOM Nancy
> > > > > Trainee at Worldline - Villeurbanne
> > > > >
> > > > > 2015-08-20 15:04 GMT+02:00 Loïc Chanel <
> loic.chanel@telecomnancy.net
> > >:
> > > > >
> > > > >> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
> > > > >> I have unlimited strength JCE installed.
> > > > >>
> > > > >> I'll try to have more clues with this option.
> > > > >>
> > > > >> Loïc CHANEL
> > > > >> Engineering student at TELECOM Nancy
> > > > >> Trainee at Worldline - Villeurbanne
> > > > >>
> > > > >> 2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
> > > > >>
> > > > >>> Which hbase / hadoop release are you using ?
> > > > >>>
> > > > >>> Running with -Dsun.security.krb5.debug=true will provide more
> clue.
> > > > >>>
> > > > >>> Do you have unlimited strength JCE installed ?
> > > > >>>
> > > > >>> Cheers
> > > > >>>
> > > > >>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
> > > > >>> loic.chanel@telecomnancy.net>
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Hi all,
> > > > >>> >
> > > > >>> > Since I kerberized my cluster, it seems like I can't use HBase
> > > > anymore
> > > > >>> ...
> > > > >>> > For example, executing create 'toto','titi' on HBase shell
> > results
> > > > in
> > > > >>> the
> > > > >>> > printing of this line endlessly :
> > > > >>> > WARN [main] security.UserGroupInformation: Not attempting to
> > > > re-login
> > > > >>> > since the last re-login was attempted less than 600 seconds
> > before.
> > > > >>> >
> > > > >>> > And nothing else happens.
> > > > >>> > I tried to restart HDFS and HBase, and to re-generate
> credentials
> > > and
> > > > >>> > keytabs, but nothing changed.
> > > > >>> > As for the logs, they are not very explicits, as the only thing
> > > they
> > > > >>> say
> > > > >>> > (and keep saying) is :
> > > > >>> >
> > > > >>> > 2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
> > > > >>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> > > > >>> > 2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
> > > > >>> > ipc.RpcServer: Have read input token of size 650 for processing
> > by
> > > > >>> > saslServer.evaluateResponse()
> > > > >>> > 2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
> > > > >>> > ipc.RpcServer: Will send token of size 108 from saslServer.
> > > > >>> > 2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
> > > > >>> > ipc.RpcServer: Have read input token of size 0 for processing
> by
> > > > >>> > saslServer.evaluateResponse()
> > > > >>> > 2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
> > > > >>> > ipc.RpcServer: Will send token of size 32 from saslServer.
> > > > >>> > 2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
> > > > >>> > ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING
> > client
> > > > >>> > 192.168.6.148:43014 because read count=-1. Number of active
> > > > >>> connections: 3
> > > > >>> >
> > > > >>> > Do anyone has an idea about where this might come from, or how
> to
> > > > >>> solve it
> > > > >>> > ? Because I couldn't find much documentation about this.
> > > > >>> > Thanks in advance for your help !
> > > > >>> >
> > > > >>> >
> > > > >>> > Loïc
> > > > >>> >
> > > > >>> > Loïc CHANEL
> > > > >>> > Engineering student at TELECOM Nancy
> > > > >>> > Trainee at Worldline - Villeurbanne
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Gupta
> > >
> >
>
Re: Problem with HBase + Kerberos
Posted by Laurent H <la...@gmail.com>.
Do you change some stuff in your hbase-site.xml when you've installed
Kerberos ?
--
Laurent HATIER - Consultant Big Data & Business Intelligence chez CapGemini
fr.linkedin.com/pub/laurent-hatier/25/36b/a86/
<http://fr.linkedin.com/pub/laurent-h/25/36b/a86/>
2015-08-21 9:44 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> Sorry if I didn't mention that, but yeah, I ran kinit before invoking hbase
> shell, and klists command says that my user has a ticket.
> [root@host /]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: testuser@REALM
>
> Valid starting Expires Service principal
> 08/21/15 09:39:33 08/22/15 09:39:33 krbtgt/REALM@REALM
> renew until 08/21/15 09:39:33
>
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-08-21 6:12 GMT+02:00 anil gupta <an...@gmail.com>:
>
> > Did you run kinit command before invoking "hbase shell"? What does klist
> > command says?
> >
> > On Thu, Aug 20, 2015 at 6:47 AM, Loïc Chanel <
> loic.chanel@telecomnancy.net
> > >
> > wrote:
> >
> > > By the way, as this may help to find my issue, I just tested typing
> > *whoami
> > > *in HBase shell : this returned me exactly what it should :
> > > testuser@REALM (auth:KERBEROS)
> > > groups: nobody, toast
> > >
> > > Loïc CHANEL
> > > Engineering student at TELECOM Nancy
> > > Trainee at Worldline - Villeurbanne
> > >
> > > 2015-08-20 15:17 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> > >
> > > > Nothing more with your option :/
> > > >
> > > > Loïc CHANEL
> > > > Engineering student at TELECOM Nancy
> > > > Trainee at Worldline - Villeurbanne
> > > >
> > > > 2015-08-20 15:04 GMT+02:00 Loïc Chanel <loic.chanel@telecomnancy.net
> >:
> > > >
> > > >> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
> > > >> I have unlimited strength JCE installed.
> > > >>
> > > >> I'll try to have more clues with this option.
> > > >>
> > > >> Loïc CHANEL
> > > >> Engineering student at TELECOM Nancy
> > > >> Trainee at Worldline - Villeurbanne
> > > >>
> > > >> 2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
> > > >>
> > > >>> Which hbase / hadoop release are you using ?
> > > >>>
> > > >>> Running with -Dsun.security.krb5.debug=true will provide more clue.
> > > >>>
> > > >>> Do you have unlimited strength JCE installed ?
> > > >>>
> > > >>> Cheers
> > > >>>
> > > >>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
> > > >>> loic.chanel@telecomnancy.net>
> > > >>> wrote:
> > > >>>
> > > >>> > Hi all,
> > > >>> >
> > > >>> > Since I kerberized my cluster, it seems like I can't use HBase
> > > anymore
> > > >>> ...
> > > >>> > For example, executing create 'toto','titi' on HBase shell
> results
> > > in
> > > >>> the
> > > >>> > printing of this line endlessly :
> > > >>> > WARN [main] security.UserGroupInformation: Not attempting to
> > > re-login
> > > >>> > since the last re-login was attempted less than 600 seconds
> before.
> > > >>> >
> > > >>> > And nothing else happens.
> > > >>> > I tried to restart HDFS and HBase, and to re-generate credentials
> > and
> > > >>> > keytabs, but nothing changed.
> > > >>> > As for the logs, they are not very explicits, as the only thing
> > they
> > > >>> say
> > > >>> > (and keep saying) is :
> > > >>> >
> > > >>> > 2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
> > > >>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> > > >>> > 2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
> > > >>> > ipc.RpcServer: Have read input token of size 650 for processing
> by
> > > >>> > saslServer.evaluateResponse()
> > > >>> > 2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
> > > >>> > ipc.RpcServer: Will send token of size 108 from saslServer.
> > > >>> > 2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
> > > >>> > ipc.RpcServer: Have read input token of size 0 for processing by
> > > >>> > saslServer.evaluateResponse()
> > > >>> > 2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
> > > >>> > ipc.RpcServer: Will send token of size 32 from saslServer.
> > > >>> > 2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
> > > >>> > ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING
> client
> > > >>> > 192.168.6.148:43014 because read count=-1. Number of active
> > > >>> connections: 3
> > > >>> >
> > > >>> > Do anyone has an idea about where this might come from, or how to
> > > >>> solve it
> > > >>> > ? Because I couldn't find much documentation about this.
> > > >>> > Thanks in advance for your help !
> > > >>> >
> > > >>> >
> > > >>> > Loïc
> > > >>> >
> > > >>> > Loïc CHANEL
> > > >>> > Engineering student at TELECOM Nancy
> > > >>> > Trainee at Worldline - Villeurbanne
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
Re: Problem with HBase + Kerberos
Posted by Loïc Chanel <lo...@telecomnancy.net>.
As someone advised me to do so, I made a kinit with the keytabs on all the
hosts where HBase component are running, and restarted HDFS and HBase, as
when restarting HBase the first time I had an error linked to hdfs keytab.
Once this is done, the errors are gone for some time, but I would bet my
paycheck that tomorrow I'll have the same errors again.
Therefore, this is most certainly related to a Kerberos expiration, but why
doesn't HBase try to renew the ticket which seems to be expired ?
As this is highly linked to Ambari deployment of Kerberos, I added the
corresponding mailing list to the discussion, hoping that someone may have
a clear idea on how to solve this problem.
Thanks in advance for your help,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-09-01 10:15 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> But how could the credentials be invalid, as they were created and managed
> only by Ambari ?
> Also I tried to connect manually with the keytab, and it works :
>
> kinit -k -t /etc/security/keytabs/hbase.service.keytab
> hbase/vm-regionserver@REALM.WL
> [root@vm-regionserver /]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: hbase/vm-regionserver@REALM.WL
>
> Valid starting Expires Service principal
> 09/01/15 10:02:18 09/02/15 10:02:18 krbtgt/REALM.WL@REALM.WL
> renew until 09/01/15 10:02:18
>
> But I still have the errors in HBase RegionServer logs :
>
> 2015-09-01 10:04:41,616 DEBUG [regionserver60020]
> security.HBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos
> principal name is hbase/vm-master@REALM.WL
> 2015-09-01 10:04:41,617 WARN [regionserver60020] ipc.RpcClient: Couldn't
> setup connection for hbase/vm-regionserver@WESTEROS.WL to
> hbase/vm-master@REALM.WL
> 2015-09-01 10:04:41,618 WARN [regionserver60020]
> regionserver.HRegionServer: error telling master we are up
> com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup
> connection for hbase/vm-regionserver@REALM.WL to hbase/vm-master@REALM.WL
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1739)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
> at
> org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:5402)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2114)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:877)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Couldn't setup connection for
> hbase/vm-regionserver@REALM.WL to hbase/vm-master@REALM.WL
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
> ... 5 more
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(Unknown
> Source)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:177)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
> ... 9 more
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Unknown
> Source)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Unknown Source)
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(Unknown
> Source)
> at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
> at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
> ... 19 more
> 2015-09-01 10:04:41,619 WARN [regionserver60020]
> regionserver.HRegionServer: reportForDuty failed; sleeping and then
> retrying.
>
> So I don't see what I could check or change to make these errors disappear.
> Is there something I'm missing ?
>
> Thanks,
>
>
> Loïc
>
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-08-31 19:20 GMT+02:00 Ted Yu <yu...@gmail.com>:
>
>> Hi,
>> The keytab you used seems to be headless keytab.
>> Here is the sample output from klist when keytab for hbase services is
>> used:
>>
>> klist
>> Ticket cache: FILE:/tmp/krb5cc_1002
>> Default principal: hbase/xxx.novalocal@EXAMPLE.COM
>>
>> Valid starting Expires Service principal
>> 31/08/2015 17:19 01/09/2015 17:19 krbtgt/EXAMPLE.COM@EXAMPLE.COM
>> renew until 31/08/2015 17:19
>>
>> FYI
>>
>> On Fri, Aug 21, 2015 at 12:44 AM, Loïc Chanel <
>> loic.chanel@telecomnancy.net>
>> wrote:
>>
>> > Sorry if I didn't mention that, but yeah, I ran kinit before invoking
>> hbase
>> > shell, and klists command says that my user has a ticket.
>> > [root@host /]# klist
>> > Ticket cache: FILE:/tmp/krb5cc_0
>> > Default principal: testuser@REALM
>> >
>> > Valid starting Expires Service principal
>> > 08/21/15 09:39:33 08/22/15 09:39:33 krbtgt/REALM@REALM
>> > renew until 08/21/15 09:39:33
>> >
>> >
>> > Loïc CHANEL
>> > Engineering student at TELECOM Nancy
>> > Trainee at Worldline - Villeurbanne
>> >
>> > 2015-08-21 6:12 GMT+02:00 anil gupta <an...@gmail.com>:
>> >
>> > > Did you run kinit command before invoking "hbase shell"? What does
>> klist
>> > > command says?
>> > >
>> > > On Thu, Aug 20, 2015 at 6:47 AM, Loïc Chanel <
>> > loic.chanel@telecomnancy.net
>> > > >
>> > > wrote:
>> > >
>> > > > By the way, as this may help to find my issue, I just tested typing
>> > > *whoami
>> > > > *in HBase shell : this returned me exactly what it should :
>> > > > testuser@REALM (auth:KERBEROS)
>> > > > groups: nobody, toast
>> > > >
>> > > > Loïc CHANEL
>> > > > Engineering student at TELECOM Nancy
>> > > > Trainee at Worldline - Villeurbanne
>> > > >
>> > > > 2015-08-20 15:17 GMT+02:00 Loïc Chanel <
>> loic.chanel@telecomnancy.net>:
>> > > >
>> > > > > Nothing more with your option :/
>> > > > >
>> > > > > Loïc CHANEL
>> > > > > Engineering student at TELECOM Nancy
>> > > > > Trainee at Worldline - Villeurbanne
>> > > > >
>> > > > > 2015-08-20 15:04 GMT+02:00 Loïc Chanel <
>> loic.chanel@telecomnancy.net
>> > >:
>> > > > >
>> > > > >> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
>> > > > >> I have unlimited strength JCE installed.
>> > > > >>
>> > > > >> I'll try to have more clues with this option.
>> > > > >>
>> > > > >> Loïc CHANEL
>> > > > >> Engineering student at TELECOM Nancy
>> > > > >> Trainee at Worldline - Villeurbanne
>> > > > >>
>> > > > >> 2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
>> > > > >>
>> > > > >>> Which hbase / hadoop release are you using ?
>> > > > >>>
>> > > > >>> Running with -Dsun.security.krb5.debug=true will provide more
>> clue.
>> > > > >>>
>> > > > >>> Do you have unlimited strength JCE installed ?
>> > > > >>>
>> > > > >>> Cheers
>> > > > >>>
>> > > > >>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
>> > > > >>> loic.chanel@telecomnancy.net>
>> > > > >>> wrote:
>> > > > >>>
>> > > > >>> > Hi all,
>> > > > >>> >
>> > > > >>> > Since I kerberized my cluster, it seems like I can't use HBase
>> > > > anymore
>> > > > >>> ...
>> > > > >>> > For example, executing create 'toto','titi' on HBase shell
>> > results
>> > > > in
>> > > > >>> the
>> > > > >>> > printing of this line endlessly :
>> > > > >>> > WARN [main] security.UserGroupInformation: Not attempting to
>> > > > re-login
>> > > > >>> > since the last re-login was attempted less than 600 seconds
>> > before.
>> > > > >>> >
>> > > > >>> > And nothing else happens.
>> > > > >>> > I tried to restart HDFS and HBase, and to re-generate
>> credentials
>> > > and
>> > > > >>> > keytabs, but nothing changed.
>> > > > >>> > As for the logs, they are not very explicits, as the only
>> thing
>> > > they
>> > > > >>> say
>> > > > >>> > (and keep saying) is :
>> > > > >>> >
>> > > > >>> > 2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
>> > > > >>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>> > > > >>> > 2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
>> > > > >>> > ipc.RpcServer: Have read input token of size 650 for
>> processing
>> > by
>> > > > >>> > saslServer.evaluateResponse()
>> > > > >>> > 2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
>> > > > >>> > ipc.RpcServer: Will send token of size 108 from saslServer.
>> > > > >>> > 2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
>> > > > >>> > ipc.RpcServer: Have read input token of size 0 for processing
>> by
>> > > > >>> > saslServer.evaluateResponse()
>> > > > >>> > 2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
>> > > > >>> > ipc.RpcServer: Will send token of size 32 from saslServer.
>> > > > >>> > 2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
>> > > > >>> > ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING
>> > client
>> > > > >>> > 192.168.6.148:43014 because read count=-1. Number of active
>> > > > >>> connections: 3
>> > > > >>> >
>> > > > >>> > Do anyone has an idea about where this might come from, or
>> how to
>> > > > >>> solve it
>> > > > >>> > ? Because I couldn't find much documentation about this.
>> > > > >>> > Thanks in advance for your help !
>> > > > >>> >
>> > > > >>> >
>> > > > >>> > Loïc
>> > > > >>> >
>> > > > >>> > Loïc CHANEL
>> > > > >>> > Engineering student at TELECOM Nancy
>> > > > >>> > Trainee at Worldline - Villeurbanne
>> > > > >>> >
>> > > > >>>
>> > > > >>
>> > > > >>
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Thanks & Regards,
>> > > Anil Gupta
>> > >
>> >
>>
>
>
Re: Problem with HBase + Kerberos
Posted by Loïc Chanel <lo...@telecomnancy.net>.
As someone advised me to do so, I made a kinit with the keytabs on all the
hosts where HBase component are running, and restarted HDFS and HBase, as
when restarting HBase the first time I had an error linked to hdfs keytab.
Once this is done, the errors are gone for some time, but I would bet my
paycheck that tomorrow I'll have the same errors again.
Therefore, this is most certainly related to a Kerberos expiration, but why
doesn't HBase try to renew the ticket which seems to be expired ?
As this is highly linked to Ambari deployment of Kerberos, I added the
corresponding mailing list to the discussion, hoping that someone may have
a clear idea on how to solve this problem.
Thanks in advance for your help,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-09-01 10:15 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> But how could the credentials be invalid, as they were created and managed
> only by Ambari ?
> Also I tried to connect manually with the keytab, and it works :
>
> kinit -k -t /etc/security/keytabs/hbase.service.keytab
> hbase/vm-regionserver@REALM.WL
> [root@vm-regionserver /]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: hbase/vm-regionserver@REALM.WL
>
> Valid starting Expires Service principal
> 09/01/15 10:02:18 09/02/15 10:02:18 krbtgt/REALM.WL@REALM.WL
> renew until 09/01/15 10:02:18
>
> But I still have the errors in HBase RegionServer logs :
>
> 2015-09-01 10:04:41,616 DEBUG [regionserver60020]
> security.HBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos
> principal name is hbase/vm-master@REALM.WL
> 2015-09-01 10:04:41,617 WARN [regionserver60020] ipc.RpcClient: Couldn't
> setup connection for hbase/vm-regionserver@WESTEROS.WL to
> hbase/vm-master@REALM.WL
> 2015-09-01 10:04:41,618 WARN [regionserver60020]
> regionserver.HRegionServer: error telling master we are up
> com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup
> connection for hbase/vm-regionserver@REALM.WL to hbase/vm-master@REALM.WL
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1739)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
> at
> org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:5402)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2114)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:877)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.IOException: Couldn't setup connection for
> hbase/vm-regionserver@REALM.WL to hbase/vm-master@REALM.WL
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
> ... 5 more
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(Unknown
> Source)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:177)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
> ... 9 more
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Unknown
> Source)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Unknown Source)
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(Unknown
> Source)
> at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
> at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
> ... 19 more
> 2015-09-01 10:04:41,619 WARN [regionserver60020]
> regionserver.HRegionServer: reportForDuty failed; sleeping and then
> retrying.
>
> So I don't see what I could check or change to make these errors disappear.
> Is there something I'm missing ?
>
> Thanks,
>
>
> Loïc
>
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-08-31 19:20 GMT+02:00 Ted Yu <yu...@gmail.com>:
>
>> Hi,
>> The keytab you used seems to be headless keytab.
>> Here is the sample output from klist when keytab for hbase services is
>> used:
>>
>> klist
>> Ticket cache: FILE:/tmp/krb5cc_1002
>> Default principal: hbase/xxx.novalocal@EXAMPLE.COM
>>
>> Valid starting Expires Service principal
>> 31/08/2015 17:19 01/09/2015 17:19 krbtgt/EXAMPLE.COM@EXAMPLE.COM
>> renew until 31/08/2015 17:19
>>
>> FYI
>>
>> On Fri, Aug 21, 2015 at 12:44 AM, Loïc Chanel <
>> loic.chanel@telecomnancy.net>
>> wrote:
>>
>> > Sorry if I didn't mention that, but yeah, I ran kinit before invoking
>> hbase
>> > shell, and klists command says that my user has a ticket.
>> > [root@host /]# klist
>> > Ticket cache: FILE:/tmp/krb5cc_0
>> > Default principal: testuser@REALM
>> >
>> > Valid starting Expires Service principal
>> > 08/21/15 09:39:33 08/22/15 09:39:33 krbtgt/REALM@REALM
>> > renew until 08/21/15 09:39:33
>> >
>> >
>> > Loïc CHANEL
>> > Engineering student at TELECOM Nancy
>> > Trainee at Worldline - Villeurbanne
>> >
>> > 2015-08-21 6:12 GMT+02:00 anil gupta <an...@gmail.com>:
>> >
>> > > Did you run kinit command before invoking "hbase shell"? What does
>> klist
>> > > command says?
>> > >
>> > > On Thu, Aug 20, 2015 at 6:47 AM, Loïc Chanel <
>> > loic.chanel@telecomnancy.net
>> > > >
>> > > wrote:
>> > >
>> > > > By the way, as this may help to find my issue, I just tested typing
>> > > *whoami
>> > > > *in HBase shell : this returned me exactly what it should :
>> > > > testuser@REALM (auth:KERBEROS)
>> > > > groups: nobody, toast
>> > > >
>> > > > Loïc CHANEL
>> > > > Engineering student at TELECOM Nancy
>> > > > Trainee at Worldline - Villeurbanne
>> > > >
>> > > > 2015-08-20 15:17 GMT+02:00 Loïc Chanel <
>> loic.chanel@telecomnancy.net>:
>> > > >
>> > > > > Nothing more with your option :/
>> > > > >
>> > > > > Loïc CHANEL
>> > > > > Engineering student at TELECOM Nancy
>> > > > > Trainee at Worldline - Villeurbanne
>> > > > >
>> > > > > 2015-08-20 15:04 GMT+02:00 Loïc Chanel <
>> loic.chanel@telecomnancy.net
>> > >:
>> > > > >
>> > > > >> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
>> > > > >> I have unlimited strength JCE installed.
>> > > > >>
>> > > > >> I'll try to have more clues with this option.
>> > > > >>
>> > > > >> Loïc CHANEL
>> > > > >> Engineering student at TELECOM Nancy
>> > > > >> Trainee at Worldline - Villeurbanne
>> > > > >>
>> > > > >> 2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
>> > > > >>
>> > > > >>> Which hbase / hadoop release are you using ?
>> > > > >>>
>> > > > >>> Running with -Dsun.security.krb5.debug=true will provide more
>> clue.
>> > > > >>>
>> > > > >>> Do you have unlimited strength JCE installed ?
>> > > > >>>
>> > > > >>> Cheers
>> > > > >>>
>> > > > >>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
>> > > > >>> loic.chanel@telecomnancy.net>
>> > > > >>> wrote:
>> > > > >>>
>> > > > >>> > Hi all,
>> > > > >>> >
>> > > > >>> > Since I kerberized my cluster, it seems like I can't use HBase
>> > > > anymore
>> > > > >>> ...
>> > > > >>> > For example, executing create 'toto','titi' on HBase shell
>> > results
>> > > > in
>> > > > >>> the
>> > > > >>> > printing of this line endlessly :
>> > > > >>> > WARN [main] security.UserGroupInformation: Not attempting to
>> > > > re-login
>> > > > >>> > since the last re-login was attempted less than 600 seconds
>> > before.
>> > > > >>> >
>> > > > >>> > And nothing else happens.
>> > > > >>> > I tried to restart HDFS and HBase, and to re-generate
>> credentials
>> > > and
>> > > > >>> > keytabs, but nothing changed.
>> > > > >>> > As for the logs, they are not very explicits, as the only
>> thing
>> > > they
>> > > > >>> say
>> > > > >>> > (and keep saying) is :
>> > > > >>> >
>> > > > >>> > 2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
>> > > > >>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>> > > > >>> > 2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
>> > > > >>> > ipc.RpcServer: Have read input token of size 650 for
>> processing
>> > by
>> > > > >>> > saslServer.evaluateResponse()
>> > > > >>> > 2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
>> > > > >>> > ipc.RpcServer: Will send token of size 108 from saslServer.
>> > > > >>> > 2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
>> > > > >>> > ipc.RpcServer: Have read input token of size 0 for processing
>> by
>> > > > >>> > saslServer.evaluateResponse()
>> > > > >>> > 2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
>> > > > >>> > ipc.RpcServer: Will send token of size 32 from saslServer.
>> > > > >>> > 2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
>> > > > >>> > ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING
>> > client
>> > > > >>> > 192.168.6.148:43014 because read count=-1. Number of active
>> > > > >>> connections: 3
>> > > > >>> >
>> > > > >>> > Do anyone has an idea about where this might come from, or
>> how to
>> > > > >>> solve it
>> > > > >>> > ? Because I couldn't find much documentation about this.
>> > > > >>> > Thanks in advance for your help !
>> > > > >>> >
>> > > > >>> >
>> > > > >>> > Loïc
>> > > > >>> >
>> > > > >>> > Loïc CHANEL
>> > > > >>> > Engineering student at TELECOM Nancy
>> > > > >>> > Trainee at Worldline - Villeurbanne
>> > > > >>> >
>> > > > >>>
>> > > > >>
>> > > > >>
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Thanks & Regards,
>> > > Anil Gupta
>> > >
>> >
>>
>
>
Re: Problem with HBase + Kerberos
Posted by Loïc Chanel <lo...@telecomnancy.net>.
But how could the credentials be invalid, as they were created and managed
only by Ambari ?
Also I tried to connect manually with the keytab, and it works :
kinit -k -t /etc/security/keytabs/hbase.service.keytab
hbase/vm-regionserver@REALM.WL
[root@vm-regionserver /]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hbase/vm-regionserver@REALM.WL
Valid starting Expires Service principal
09/01/15 10:02:18 09/02/15 10:02:18 krbtgt/REALM.WL@REALM.WL
renew until 09/01/15 10:02:18
But I still have the errors in HBase RegionServer logs :
2015-09-01 10:04:41,616 DEBUG [regionserver60020]
security.HBaseSaslRpcClient: Creating SASL GSSAPI client. Server's Kerberos
principal name is hbase/vm-master@REALM.WL
2015-09-01 10:04:41,617 WARN [regionserver60020] ipc.RpcClient: Couldn't
setup connection for hbase/vm-regionserver@WESTEROS.WL to
hbase/vm-master@REALM.WL
2015-09-01 10:04:41,618 WARN [regionserver60020]
regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup
connection for hbase/vm-regionserver@REALM.WL to hbase/vm-master@REALM.WL
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1739)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
at
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:5402)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2114)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:877)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Couldn't setup connection for
hbase/vm-regionserver@REALM.WL to hbase/vm-master@REALM.WL
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
... 5 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused
by GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(Unknown
Source)
at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:177)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
... 9 more
Caused by: GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Unknown
Source)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Unknown Source)
at
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Unknown Source)
at sun.security.jgss.GSSManagerImpl.getMechanismContext(Unknown
Source)
at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
at sun.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
... 19 more
2015-09-01 10:04:41,619 WARN [regionserver60020]
regionserver.HRegionServer: reportForDuty failed; sleeping and then
retrying.
So I don't see what I could check or change to make these errors disappear.
Is there something I'm missing ?
Thanks,
Loïc
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-08-31 19:20 GMT+02:00 Ted Yu <yu...@gmail.com>:
> Hi,
> The keytab you used seems to be headless keytab.
> Here is the sample output from klist when keytab for hbase services is
> used:
>
> klist
> Ticket cache: FILE:/tmp/krb5cc_1002
> Default principal: hbase/xxx.novalocal@EXAMPLE.COM
>
> Valid starting Expires Service principal
> 31/08/2015 17:19 01/09/2015 17:19 krbtgt/EXAMPLE.COM@EXAMPLE.COM
> renew until 31/08/2015 17:19
>
> FYI
>
> On Fri, Aug 21, 2015 at 12:44 AM, Loïc Chanel <
> loic.chanel@telecomnancy.net>
> wrote:
>
> > Sorry if I didn't mention that, but yeah, I ran kinit before invoking
> hbase
> > shell, and klists command says that my user has a ticket.
> > [root@host /]# klist
> > Ticket cache: FILE:/tmp/krb5cc_0
> > Default principal: testuser@REALM
> >
> > Valid starting Expires Service principal
> > 08/21/15 09:39:33 08/22/15 09:39:33 krbtgt/REALM@REALM
> > renew until 08/21/15 09:39:33
> >
> >
> > Loïc CHANEL
> > Engineering student at TELECOM Nancy
> > Trainee at Worldline - Villeurbanne
> >
> > 2015-08-21 6:12 GMT+02:00 anil gupta <an...@gmail.com>:
> >
> > > Did you run kinit command before invoking "hbase shell"? What does
> klist
> > > command says?
> > >
> > > On Thu, Aug 20, 2015 at 6:47 AM, Loïc Chanel <
> > loic.chanel@telecomnancy.net
> > > >
> > > wrote:
> > >
> > > > By the way, as this may help to find my issue, I just tested typing
> > > *whoami
> > > > *in HBase shell : this returned me exactly what it should :
> > > > testuser@REALM (auth:KERBEROS)
> > > > groups: nobody, toast
> > > >
> > > > Loïc CHANEL
> > > > Engineering student at TELECOM Nancy
> > > > Trainee at Worldline - Villeurbanne
> > > >
> > > > 2015-08-20 15:17 GMT+02:00 Loïc Chanel <loic.chanel@telecomnancy.net
> >:
> > > >
> > > > > Nothing more with your option :/
> > > > >
> > > > > Loïc CHANEL
> > > > > Engineering student at TELECOM Nancy
> > > > > Trainee at Worldline - Villeurbanne
> > > > >
> > > > > 2015-08-20 15:04 GMT+02:00 Loïc Chanel <
> loic.chanel@telecomnancy.net
> > >:
> > > > >
> > > > >> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
> > > > >> I have unlimited strength JCE installed.
> > > > >>
> > > > >> I'll try to have more clues with this option.
> > > > >>
> > > > >> Loïc CHANEL
> > > > >> Engineering student at TELECOM Nancy
> > > > >> Trainee at Worldline - Villeurbanne
> > > > >>
> > > > >> 2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
> > > > >>
> > > > >>> Which hbase / hadoop release are you using ?
> > > > >>>
> > > > >>> Running with -Dsun.security.krb5.debug=true will provide more
> clue.
> > > > >>>
> > > > >>> Do you have unlimited strength JCE installed ?
> > > > >>>
> > > > >>> Cheers
> > > > >>>
> > > > >>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
> > > > >>> loic.chanel@telecomnancy.net>
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Hi all,
> > > > >>> >
> > > > >>> > Since I kerberized my cluster, it seems like I can't use HBase
> > > > anymore
> > > > >>> ...
> > > > >>> > For example, executing create 'toto','titi' on HBase shell
> > results
> > > > in
> > > > >>> the
> > > > >>> > printing of this line endlessly :
> > > > >>> > WARN [main] security.UserGroupInformation: Not attempting to
> > > > re-login
> > > > >>> > since the last re-login was attempted less than 600 seconds
> > before.
> > > > >>> >
> > > > >>> > And nothing else happens.
> > > > >>> > I tried to restart HDFS and HBase, and to re-generate
> credentials
> > > and
> > > > >>> > keytabs, but nothing changed.
> > > > >>> > As for the logs, they are not very explicits, as the only thing
> > > they
> > > > >>> say
> > > > >>> > (and keep saying) is :
> > > > >>> >
> > > > >>> > 2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
> > > > >>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> > > > >>> > 2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
> > > > >>> > ipc.RpcServer: Have read input token of size 650 for processing
> > by
> > > > >>> > saslServer.evaluateResponse()
> > > > >>> > 2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
> > > > >>> > ipc.RpcServer: Will send token of size 108 from saslServer.
> > > > >>> > 2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
> > > > >>> > ipc.RpcServer: Have read input token of size 0 for processing
> by
> > > > >>> > saslServer.evaluateResponse()
> > > > >>> > 2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
> > > > >>> > ipc.RpcServer: Will send token of size 32 from saslServer.
> > > > >>> > 2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
> > > > >>> > ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING
> > client
> > > > >>> > 192.168.6.148:43014 because read count=-1. Number of active
> > > > >>> connections: 3
> > > > >>> >
> > > > >>> > Do anyone has an idea about where this might come from, or how
> to
> > > > >>> solve it
> > > > >>> > ? Because I couldn't find much documentation about this.
> > > > >>> > Thanks in advance for your help !
> > > > >>> >
> > > > >>> >
> > > > >>> > Loïc
> > > > >>> >
> > > > >>> > Loïc CHANEL
> > > > >>> > Engineering student at TELECOM Nancy
> > > > >>> > Trainee at Worldline - Villeurbanne
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks & Regards,
> > > Anil Gupta
> > >
> >
>
Re: Problem with HBase + Kerberos
Posted by Ted Yu <yu...@gmail.com>.
Hi,
The keytab you used seems to be headless keytab.
Here is the sample output from klist when keytab for hbase services is used:
klist
Ticket cache: FILE:/tmp/krb5cc_1002
Default principal: hbase/xxx.novalocal@EXAMPLE.COM
Valid starting Expires Service principal
31/08/2015 17:19 01/09/2015 17:19 krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 31/08/2015 17:19
FYI
On Fri, Aug 21, 2015 at 12:44 AM, Loïc Chanel <lo...@telecomnancy.net>
wrote:
> Sorry if I didn't mention that, but yeah, I ran kinit before invoking hbase
> shell, and klists command says that my user has a ticket.
> [root@host /]# klist
> Ticket cache: FILE:/tmp/krb5cc_0
> Default principal: testuser@REALM
>
> Valid starting Expires Service principal
> 08/21/15 09:39:33 08/22/15 09:39:33 krbtgt/REALM@REALM
> renew until 08/21/15 09:39:33
>
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-08-21 6:12 GMT+02:00 anil gupta <an...@gmail.com>:
>
> > Did you run kinit command before invoking "hbase shell"? What does klist
> > command says?
> >
> > On Thu, Aug 20, 2015 at 6:47 AM, Loïc Chanel <
> loic.chanel@telecomnancy.net
> > >
> > wrote:
> >
> > > By the way, as this may help to find my issue, I just tested typing
> > *whoami
> > > *in HBase shell : this returned me exactly what it should :
> > > testuser@REALM (auth:KERBEROS)
> > > groups: nobody, toast
> > >
> > > Loïc CHANEL
> > > Engineering student at TELECOM Nancy
> > > Trainee at Worldline - Villeurbanne
> > >
> > > 2015-08-20 15:17 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> > >
> > > > Nothing more with your option :/
> > > >
> > > > Loïc CHANEL
> > > > Engineering student at TELECOM Nancy
> > > > Trainee at Worldline - Villeurbanne
> > > >
> > > > 2015-08-20 15:04 GMT+02:00 Loïc Chanel <loic.chanel@telecomnancy.net
> >:
> > > >
> > > >> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
> > > >> I have unlimited strength JCE installed.
> > > >>
> > > >> I'll try to have more clues with this option.
> > > >>
> > > >> Loïc CHANEL
> > > >> Engineering student at TELECOM Nancy
> > > >> Trainee at Worldline - Villeurbanne
> > > >>
> > > >> 2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
> > > >>
> > > >>> Which hbase / hadoop release are you using ?
> > > >>>
> > > >>> Running with -Dsun.security.krb5.debug=true will provide more clue.
> > > >>>
> > > >>> Do you have unlimited strength JCE installed ?
> > > >>>
> > > >>> Cheers
> > > >>>
> > > >>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
> > > >>> loic.chanel@telecomnancy.net>
> > > >>> wrote:
> > > >>>
> > > >>> > Hi all,
> > > >>> >
> > > >>> > Since I kerberized my cluster, it seems like I can't use HBase
> > > anymore
> > > >>> ...
> > > >>> > For example, executing create 'toto','titi' on HBase shell
> results
> > > in
> > > >>> the
> > > >>> > printing of this line endlessly :
> > > >>> > WARN [main] security.UserGroupInformation: Not attempting to
> > > re-login
> > > >>> > since the last re-login was attempted less than 600 seconds
> before.
> > > >>> >
> > > >>> > And nothing else happens.
> > > >>> > I tried to restart HDFS and HBase, and to re-generate credentials
> > and
> > > >>> > keytabs, but nothing changed.
> > > >>> > As for the logs, they are not very explicits, as the only thing
> > they
> > > >>> say
> > > >>> > (and keep saying) is :
> > > >>> >
> > > >>> > 2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
> > > >>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> > > >>> > 2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
> > > >>> > ipc.RpcServer: Have read input token of size 650 for processing
> by
> > > >>> > saslServer.evaluateResponse()
> > > >>> > 2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
> > > >>> > ipc.RpcServer: Will send token of size 108 from saslServer.
> > > >>> > 2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
> > > >>> > ipc.RpcServer: Have read input token of size 0 for processing by
> > > >>> > saslServer.evaluateResponse()
> > > >>> > 2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
> > > >>> > ipc.RpcServer: Will send token of size 32 from saslServer.
> > > >>> > 2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
> > > >>> > ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING
> client
> > > >>> > 192.168.6.148:43014 because read count=-1. Number of active
> > > >>> connections: 3
> > > >>> >
> > > >>> > Do anyone has an idea about where this might come from, or how to
> > > >>> solve it
> > > >>> > ? Because I couldn't find much documentation about this.
> > > >>> > Thanks in advance for your help !
> > > >>> >
> > > >>> >
> > > >>> > Loïc
> > > >>> >
> > > >>> > Loïc CHANEL
> > > >>> > Engineering student at TELECOM Nancy
> > > >>> > Trainee at Worldline - Villeurbanne
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> >
>
Re: Problem with HBase + Kerberos
Posted by Loïc Chanel <lo...@telecomnancy.net>.
Sorry if I didn't mention that, but yeah, I ran kinit before invoking hbase
shell, and klists command says that my user has a ticket.
[root@host /]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: testuser@REALM
Valid starting Expires Service principal
08/21/15 09:39:33 08/22/15 09:39:33 krbtgt/REALM@REALM
renew until 08/21/15 09:39:33
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-08-21 6:12 GMT+02:00 anil gupta <an...@gmail.com>:
> Did you run kinit command before invoking "hbase shell"? What does klist
> command says?
>
> On Thu, Aug 20, 2015 at 6:47 AM, Loïc Chanel <loic.chanel@telecomnancy.net
> >
> wrote:
>
> > By the way, as this may help to find my issue, I just tested typing
> *whoami
> > *in HBase shell : this returned me exactly what it should :
> > testuser@REALM (auth:KERBEROS)
> > groups: nobody, toast
> >
> > Loïc CHANEL
> > Engineering student at TELECOM Nancy
> > Trainee at Worldline - Villeurbanne
> >
> > 2015-08-20 15:17 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> >
> > > Nothing more with your option :/
> > >
> > > Loïc CHANEL
> > > Engineering student at TELECOM Nancy
> > > Trainee at Worldline - Villeurbanne
> > >
> > > 2015-08-20 15:04 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> > >
> > >> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
> > >> I have unlimited strength JCE installed.
> > >>
> > >> I'll try to have more clues with this option.
> > >>
> > >> Loïc CHANEL
> > >> Engineering student at TELECOM Nancy
> > >> Trainee at Worldline - Villeurbanne
> > >>
> > >> 2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
> > >>
> > >>> Which hbase / hadoop release are you using ?
> > >>>
> > >>> Running with -Dsun.security.krb5.debug=true will provide more clue.
> > >>>
> > >>> Do you have unlimited strength JCE installed ?
> > >>>
> > >>> Cheers
> > >>>
> > >>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
> > >>> loic.chanel@telecomnancy.net>
> > >>> wrote:
> > >>>
> > >>> > Hi all,
> > >>> >
> > >>> > Since I kerberized my cluster, it seems like I can't use HBase
> > anymore
> > >>> ...
> > >>> > For example, executing create 'toto','titi' on HBase shell results
> > in
> > >>> the
> > >>> > printing of this line endlessly :
> > >>> > WARN [main] security.UserGroupInformation: Not attempting to
> > re-login
> > >>> > since the last re-login was attempted less than 600 seconds before.
> > >>> >
> > >>> > And nothing else happens.
> > >>> > I tried to restart HDFS and HBase, and to re-generate credentials
> and
> > >>> > keytabs, but nothing changed.
> > >>> > As for the logs, they are not very explicits, as the only thing
> they
> > >>> say
> > >>> > (and keep saying) is :
> > >>> >
> > >>> > 2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
> > >>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> > >>> > 2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
> > >>> > ipc.RpcServer: Have read input token of size 650 for processing by
> > >>> > saslServer.evaluateResponse()
> > >>> > 2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
> > >>> > ipc.RpcServer: Will send token of size 108 from saslServer.
> > >>> > 2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
> > >>> > ipc.RpcServer: Have read input token of size 0 for processing by
> > >>> > saslServer.evaluateResponse()
> > >>> > 2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
> > >>> > ipc.RpcServer: Will send token of size 32 from saslServer.
> > >>> > 2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
> > >>> > ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING client
> > >>> > 192.168.6.148:43014 because read count=-1. Number of active
> > >>> connections: 3
> > >>> >
> > >>> > Do anyone has an idea about where this might come from, or how to
> > >>> solve it
> > >>> > ? Because I couldn't find much documentation about this.
> > >>> > Thanks in advance for your help !
> > >>> >
> > >>> >
> > >>> > Loïc
> > >>> >
> > >>> > Loïc CHANEL
> > >>> > Engineering student at TELECOM Nancy
> > >>> > Trainee at Worldline - Villeurbanne
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
Re: Problem with HBase + Kerberos
Posted by anil gupta <an...@gmail.com>.
Did you run kinit command before invoking "hbase shell"? What does klist
command says?
On Thu, Aug 20, 2015 at 6:47 AM, Loïc Chanel <lo...@telecomnancy.net>
wrote:
> By the way, as this may help to find my issue, I just tested typing *whoami
> *in HBase shell : this returned me exactly what it should :
> testuser@REALM (auth:KERBEROS)
> groups: nobody, toast
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-08-20 15:17 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>
> > Nothing more with your option :/
> >
> > Loïc CHANEL
> > Engineering student at TELECOM Nancy
> > Trainee at Worldline - Villeurbanne
> >
> > 2015-08-20 15:04 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> >
> >> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
> >> I have unlimited strength JCE installed.
> >>
> >> I'll try to have more clues with this option.
> >>
> >> Loïc CHANEL
> >> Engineering student at TELECOM Nancy
> >> Trainee at Worldline - Villeurbanne
> >>
> >> 2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
> >>
> >>> Which hbase / hadoop release are you using ?
> >>>
> >>> Running with -Dsun.security.krb5.debug=true will provide more clue.
> >>>
> >>> Do you have unlimited strength JCE installed ?
> >>>
> >>> Cheers
> >>>
> >>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
> >>> loic.chanel@telecomnancy.net>
> >>> wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > Since I kerberized my cluster, it seems like I can't use HBase
> anymore
> >>> ...
> >>> > For example, executing create 'toto','titi' on HBase shell results
> in
> >>> the
> >>> > printing of this line endlessly :
> >>> > WARN [main] security.UserGroupInformation: Not attempting to
> re-login
> >>> > since the last re-login was attempted less than 600 seconds before.
> >>> >
> >>> > And nothing else happens.
> >>> > I tried to restart HDFS and HBase, and to re-generate credentials and
> >>> > keytabs, but nothing changed.
> >>> > As for the logs, they are not very explicits, as the only thing they
> >>> say
> >>> > (and keep saying) is :
> >>> >
> >>> > 2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
> >>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> >>> > 2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
> >>> > ipc.RpcServer: Have read input token of size 650 for processing by
> >>> > saslServer.evaluateResponse()
> >>> > 2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
> >>> > ipc.RpcServer: Will send token of size 108 from saslServer.
> >>> > 2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
> >>> > ipc.RpcServer: Have read input token of size 0 for processing by
> >>> > saslServer.evaluateResponse()
> >>> > 2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
> >>> > ipc.RpcServer: Will send token of size 32 from saslServer.
> >>> > 2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
> >>> > ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING client
> >>> > 192.168.6.148:43014 because read count=-1. Number of active
> >>> connections: 3
> >>> >
> >>> > Do anyone has an idea about where this might come from, or how to
> >>> solve it
> >>> > ? Because I couldn't find much documentation about this.
> >>> > Thanks in advance for your help !
> >>> >
> >>> >
> >>> > Loïc
> >>> >
> >>> > Loïc CHANEL
> >>> > Engineering student at TELECOM Nancy
> >>> > Trainee at Worldline - Villeurbanne
> >>> >
> >>>
> >>
> >>
> >
>
--
Thanks & Regards,
Anil Gupta
Re: Problem with HBase + Kerberos
Posted by Loïc Chanel <lo...@telecomnancy.net>.
By the way, as this may help to find my issue, I just tested typing *whoami
*in HBase shell : this returned me exactly what it should :
testuser@REALM (auth:KERBEROS)
groups: nobody, toast
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-08-20 15:17 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> Nothing more with your option :/
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-08-20 15:04 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
>
>> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
>> I have unlimited strength JCE installed.
>>
>> I'll try to have more clues with this option.
>>
>> Loïc CHANEL
>> Engineering student at TELECOM Nancy
>> Trainee at Worldline - Villeurbanne
>>
>> 2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
>>
>>> Which hbase / hadoop release are you using ?
>>>
>>> Running with -Dsun.security.krb5.debug=true will provide more clue.
>>>
>>> Do you have unlimited strength JCE installed ?
>>>
>>> Cheers
>>>
>>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
>>> loic.chanel@telecomnancy.net>
>>> wrote:
>>>
>>> > Hi all,
>>> >
>>> > Since I kerberized my cluster, it seems like I can't use HBase anymore
>>> ...
>>> > For example, executing create 'toto','titi' on HBase shell results in
>>> the
>>> > printing of this line endlessly :
>>> > WARN [main] security.UserGroupInformation: Not attempting to re-login
>>> > since the last re-login was attempted less than 600 seconds before.
>>> >
>>> > And nothing else happens.
>>> > I tried to restart HDFS and HBase, and to re-generate credentials and
>>> > keytabs, but nothing changed.
>>> > As for the logs, they are not very explicits, as the only thing they
>>> say
>>> > (and keep saying) is :
>>> >
>>> > 2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
>>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>>> > 2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
>>> > ipc.RpcServer: Have read input token of size 650 for processing by
>>> > saslServer.evaluateResponse()
>>> > 2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
>>> > ipc.RpcServer: Will send token of size 108 from saslServer.
>>> > 2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
>>> > ipc.RpcServer: Have read input token of size 0 for processing by
>>> > saslServer.evaluateResponse()
>>> > 2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
>>> > ipc.RpcServer: Will send token of size 32 from saslServer.
>>> > 2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
>>> > ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING client
>>> > 192.168.6.148:43014 because read count=-1. Number of active
>>> connections: 3
>>> >
>>> > Do anyone has an idea about where this might come from, or how to
>>> solve it
>>> > ? Because I couldn't find much documentation about this.
>>> > Thanks in advance for your help !
>>> >
>>> >
>>> > Loïc
>>> >
>>> > Loïc CHANEL
>>> > Engineering student at TELECOM Nancy
>>> > Trainee at Worldline - Villeurbanne
>>> >
>>>
>>
>>
>
Re: Problem with HBase + Kerberos
Posted by Loïc Chanel <lo...@telecomnancy.net>.
Nothing more with your option :/
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-08-20 15:04 GMT+02:00 Loïc Chanel <lo...@telecomnancy.net>:
> I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
> I have unlimited strength JCE installed.
>
> I'll try to have more clues with this option.
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>
> 2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
>
>> Which hbase / hadoop release are you using ?
>>
>> Running with -Dsun.security.krb5.debug=true will provide more clue.
>>
>> Do you have unlimited strength JCE installed ?
>>
>> Cheers
>>
>> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <
>> loic.chanel@telecomnancy.net>
>> wrote:
>>
>> > Hi all,
>> >
>> > Since I kerberized my cluster, it seems like I can't use HBase anymore
>> ...
>> > For example, executing create 'toto','titi' on HBase shell results in
>> the
>> > printing of this line endlessly :
>> > WARN [main] security.UserGroupInformation: Not attempting to re-login
>> > since the last re-login was attempted less than 600 seconds before.
>> >
>> > And nothing else happens.
>> > I tried to restart HDFS and HBase, and to re-generate credentials and
>> > keytabs, but nothing changed.
>> > As for the logs, they are not very explicits, as the only thing they say
>> > (and keep saying) is :
>> >
>> > 2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>> > 2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
>> > ipc.RpcServer: Have read input token of size 650 for processing by
>> > saslServer.evaluateResponse()
>> > 2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
>> > ipc.RpcServer: Will send token of size 108 from saslServer.
>> > 2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
>> > ipc.RpcServer: Have read input token of size 0 for processing by
>> > saslServer.evaluateResponse()
>> > 2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
>> > ipc.RpcServer: Will send token of size 32 from saslServer.
>> > 2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
>> > ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING client
>> > 192.168.6.148:43014 because read count=-1. Number of active
>> connections: 3
>> >
>> > Do anyone has an idea about where this might come from, or how to solve
>> it
>> > ? Because I couldn't find much documentation about this.
>> > Thanks in advance for your help !
>> >
>> >
>> > Loïc
>> >
>> > Loïc CHANEL
>> > Engineering student at TELECOM Nancy
>> > Trainee at Worldline - Villeurbanne
>> >
>>
>
>
Re: Problem with HBase + Kerberos
Posted by Loïc Chanel <lo...@telecomnancy.net>.
I'm using HDP 2.2.4.2, with HBase 0.98.4.2.2.
I have unlimited strength JCE installed.
I'll try to have more clues with this option.
Loïc CHANEL
Engineering student at TELECOM Nancy
Trainee at Worldline - Villeurbanne
2015-08-20 14:58 GMT+02:00 Ted Yu <yu...@gmail.com>:
> Which hbase / hadoop release are you using ?
>
> Running with -Dsun.security.krb5.debug=true will provide more clue.
>
> Do you have unlimited strength JCE installed ?
>
> Cheers
>
> On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <loic.chanel@telecomnancy.net
> >
> wrote:
>
> > Hi all,
> >
> > Since I kerberized my cluster, it seems like I can't use HBase anymore
> ...
> > For example, executing create 'toto','titi' on HBase shell results in
> the
> > printing of this line endlessly :
> > WARN [main] security.UserGroupInformation: Not attempting to re-login
> > since the last re-login was attempted less than 600 seconds before.
> >
> > And nothing else happens.
> > I tried to restart HDFS and HBase, and to re-generate credentials and
> > keytabs, but nothing changed.
> > As for the logs, they are not very explicits, as the only thing they say
> > (and keep saying) is :
> >
> > 2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> > 2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
> > ipc.RpcServer: Have read input token of size 650 for processing by
> > saslServer.evaluateResponse()
> > 2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
> > ipc.RpcServer: Will send token of size 108 from saslServer.
> > 2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
> > ipc.RpcServer: Have read input token of size 0 for processing by
> > saslServer.evaluateResponse()
> > 2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
> > ipc.RpcServer: Will send token of size 32 from saslServer.
> > 2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
> > ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING client
> > 192.168.6.148:43014 because read count=-1. Number of active
> connections: 3
> >
> > Do anyone has an idea about where this might come from, or how to solve
> it
> > ? Because I couldn't find much documentation about this.
> > Thanks in advance for your help !
> >
> >
> > Loïc
> >
> > Loïc CHANEL
> > Engineering student at TELECOM Nancy
> > Trainee at Worldline - Villeurbanne
> >
>
Re: Problem with HBase + Kerberos
Posted by Ted Yu <yu...@gmail.com>.
Which hbase / hadoop release are you using ?
Running with -Dsun.security.krb5.debug=true will provide more clue.
Do you have unlimited strength JCE installed ?
Cheers
On Thu, Aug 20, 2015 at 5:46 AM, Loïc Chanel <lo...@telecomnancy.net>
wrote:
> Hi all,
>
> Since I kerberized my cluster, it seems like I can't use HBase anymore ...
> For example, executing create 'toto','titi' on HBase shell results in the
> printing of this line endlessly :
> WARN [main] security.UserGroupInformation: Not attempting to re-login
> since the last re-login was attempted less than 600 seconds before.
>
> And nothing else happens.
> I tried to restart HDFS and HBase, and to re-generate credentials and
> keytabs, but nothing changed.
> As for the logs, they are not very explicits, as the only thing they say
> (and keep saying) is :
>
> 2015-08-20 13:50:12,697 DEBUG [RpcServer.reader=2,port=60000]
> ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> 2015-08-20 13:50:12,698 DEBUG [RpcServer.reader=2,port=60000]
> ipc.RpcServer: Have read input token of size 650 for processing by
> saslServer.evaluateResponse()
> 2015-08-20 13:50:12,704 DEBUG [RpcServer.reader=2,port=60000]
> ipc.RpcServer: Will send token of size 108 from saslServer.
> 2015-08-20 13:50:12,706 DEBUG [RpcServer.reader=2,port=60000]
> ipc.RpcServer: Have read input token of size 0 for processing by
> saslServer.evaluateResponse()
> 2015-08-20 13:50:12,707 DEBUG [RpcServer.reader=2,port=60000]
> ipc.RpcServer: Will send token of size 32 from saslServer.
> 2015-08-20 13:50:12,708 DEBUG [RpcServer.reader=2,port=60000]
> ipc.RpcServer: RpcServer.listener,port=60000: DISCONNECTING client
> 192.168.6.148:43014 because read count=-1. Number of active connections: 3
>
> Do anyone has an idea about where this might come from, or how to solve it
> ? Because I couldn't find much documentation about this.
> Thanks in advance for your help !
>
>
> Loïc
>
> Loïc CHANEL
> Engineering student at TELECOM Nancy
> Trainee at Worldline - Villeurbanne
>