You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Manoj Murumkar <ma...@gmail.com> on 2015/03/14 07:28:28 UTC
Kerberos exception
Hi,
We have a secured cluster. All components are working well, except hbase.
Specifically, this is what I see on regionserver:
2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
ipc.RpcServer: Kerberos principal name is hbase/
sfdvgctsn001.xxxxx@SFDVGCT.COM
2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
ipc.RpcServer: Created SASL server with mechanism = GSSAPI
2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
ipc.RpcServer: Have read input token of size 627 for processing by
saslServer.evaluateResponse()
2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
reading:GSS initiate fail
and on the master:
java.io.IOException: Couldn't setup connection for hbase/sfdvgctmn004.
xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
All the keytabs are set properly. Has anyone seen this before? Appreciate
the time.
Manoj
Re: Kerberos exception
Posted by Manoj Murumkar <ma...@gmail.com>.
Btw, ignore ganglia exceptions. We don't have that configured.
On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <ma...@gmail.com>
wrote:
> Attached both files.
>
>
>
> On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <ol...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Traces (especially one for region server) look a bit incomplete, did
>> you copy them fully?
>>
>> Also may help if you post relevant pieces of hbase-site.xml (with
>> security configs).
>>
>> Thanks,
>> Mikhail
>>
>> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
>> <ma...@gmail.com> wrote:
>> > Hi,
>> >
>> > We have a secured cluster. All components are working well, except
>> hbase.
>> > Specifically, this is what I see on regionserver:
>> >
>> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
>> > ipc.RpcServer: Kerberos principal name is hbase/
>> > sfdvgctsn001.xxxxx@SFDVGCT.COM
>> >
>> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>> >
>> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> > ipc.RpcServer: Have read input token of size 627 for processing by
>> > saslServer.evaluateResponse()
>> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
>> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
>> > reading:GSS initiate fail
>> >
>> >
>> > and on the master:
>> >
>> >
>> > java.io.IOException: Couldn't setup connection for hbase/sfdvgctmn004.
>> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>> >
>> > at java.security.AccessController.doPrivileged(Native Method)
>> >
>> > at javax.security.auth.Subject.doAs(Subject.java:415)
>> >
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>> >
>> > All the keytabs are set properly. Has anyone seen this before?
>> Appreciate
>> > the time.
>> >
>> > Manoj
>>
>>
>>
>> --
>> Thanks,
>> Michael Antonov
>>
>
>
Re: Kerberos exception
Posted by Ted Yu <yu...@gmail.com>.
Looks like region server log didn't go through.
Can you use pastebin ?
On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar <ma...@gmail.com>
wrote:
> Attached region server log again.
>
> Ted,
>
> Corresponding log for master is:
>
> java.io.IOException: Couldn't setup connection for hbase/
> sfdvgctmn001.gid.gap.com@SFDVGCT.COM to hbase/sfdvgctsn001.gid.gap.com@S
>
> FDVGCT.COM
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
>
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176)
>
> at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743)
>
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> GSS initiate failed
>
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)
>
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
>
> On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> bq. Attached both files
>>
>> I only found hbase-site.xml in the attachment.
>>
>> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <manoj.murumkar@gmail.com
>> >
>> wrote:
>>
>> > Attached both files.
>> >
>> >
>> >
>> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <ol...@gmail.com>
>> > wrote:
>> >
>> >> Hi,
>> >>
>> >> Traces (especially one for region server) look a bit incomplete, did
>> >> you copy them fully?
>> >>
>> >> Also may help if you post relevant pieces of hbase-site.xml (with
>> >> security configs).
>> >>
>> >> Thanks,
>> >> Mikhail
>> >>
>> >>
>> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
>> >> <ma...@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > We have a secured cluster. All components are working well, except
>> >> hbase.
>> >> > Specifically, this is what I see on regionserver:
>> >> >
>> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
>> >> > ipc.RpcServer: Kerberos principal name is hbase/
>> >> > sfdvgctsn001.xxxxx@SFDVGCT.COM
>> >> >
>> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>> >> >
>> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> >> > ipc.RpcServer: Have read input token of size 627 for processing by
>> >> > saslServer.evaluateResponse()
>> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
>> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
>> >> > reading:GSS initiate fail
>> >> >
>> >> >
>> >> > and on the master:
>> >> >
>> >> >
>> >> > java.io.IOException: Couldn't setup connection for
>> hbase/sfdvgctmn004.
>> >> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>> >> >
>> >> > at java.security.AccessController.doPrivileged(Native Method)
>> >> >
>> >> > at javax.security.auth.Subject.doAs(Subject.java:415)
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>> >> >
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>> >> >
>> >> > All the keytabs are set properly. Has anyone seen this before?
>> >> Appreciate
>> >> > the time.
>> >> >
>> >> > Manoj
>> >>
>> >>
>> >>
>> >> --
>> >> Thanks,
>> >> Michael Antonov
>> >>
>> >
>> >
>>
>
>
Re: Kerberos exception
Posted by Manoj Murumkar <ma...@gmail.com>.
I did that, to no avail. I am surprised by level of verbosity (lack
thereof) too, something new you learn every day :)
On Mar 16, 2015 3:44 PM, "Mikhail Antonov" <ol...@gmail.com> wrote:
> Glad it helped!
>
> In general though, in this cases, if you turn on java kerberos debug
> logging (via something like export
> HADOOP_OPTS="-Dsun.security.krb5.debug=true"), in your log you would
> see error like: "unsupported key type found the default TGT: 18"
>
> (which means encryption type 18 per kerberos specification, i.e.
> aes256-cts-hmac-sha1-96, isn't supported by Java - due to lack of
> unlimited strength JCE)
>
> -Mikhail
>
> On Mon, Mar 16, 2015 at 3:19 PM, Manoj Murumkar
> <ma...@gmail.com> wrote:
> > Thanks Mikhail! It turned out not having unlimited strength JCE files
> > everywhere on the cluster was root cause (I reproduced it by leaving
> > default files on regionservers on the newly built cluster).
> >
> > On Sun, Mar 15, 2015 at 9:48 PM, Manoj Murumkar <
> manoj.murumkar@gmail.com>
> > wrote:
> >
> >> Thanks. We took care of above items (all principals were generated using
> >> Ambari tool) although I am not sure about 3rd item. One of my coworkers
> >> decided to reinstall HBase, so I am waiting for the cluster to come up
> to
> >> see if we still have the issue. Funny thing is, everything else worked
> in
> >> secure environment (MR, Hive), just not HBase. The error message didn't
> >> mention any details why the GSSException was raised (sample below):
> >>
> >>
> >> 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
> >> ipc.RpcServer: Kerberos principal name is hbase/
> >> sfdvgctsn001.xxxxx@SFDVGCT.COM
> >>
> >> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >> ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> >>
> >> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >> ipc.RpcServer: Have read input token of size 627 for processing by
> >> saslServer.evaluateResponse()
> >> *2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
> >> ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
> >> reading:GSS initiate fail*
> >>
> >>
> >> On Sun, Mar 15, 2015 at 2:43 PM, Mikhail Antonov <ol...@gmail.com>
> >> wrote:
> >>
> >>> I don't see region server log file in attachment for some reason.. May
> >>> be the file is too big and gets rejected by mail server? Without it
> >>> it'd be hard to say. Also master log looks a bit incomplete, are there
> >>> no more traces? "GSS initiate fail" is normally followed by something
> >>> like (just for example) - "[Caused by GSSException: No valid
> >>> credentials provided (Mechanism level: Failed to find any Kerberos
> >>> tgt)];" or so. Nothing like that?
> >>>
> >>> But in general, GSS initialization most often fails for following
> reasons:
> >>>
> >>> - wrong linux path or file permissions on keytab file
> >>> - typo in kerberos principal/realm (that includes lower/upper case
> >>> differences)
> >>> - JCE unlimited strength file aren't installed
> >>>
> >>> On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar
> >>> <ma...@gmail.com> wrote:
> >>> > Attached region server log again.
> >>> >
> >>> > Ted,
> >>> >
> >>> > Corresponding log for master is:
> >>> >
> >>> > java.io.IOException: Couldn't setup connection for
> >>> > hbase/sfdvgctmn001.gid.gap.com@SFDVGCT.COM to
> >>> > hbase/sfdvgctsn001.gid.gap.com@S
> >>> >
> >>> > FDVGCT.COM
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
> >>> >
> >>> > at java.security.AccessController.doPrivileged(Native Method)
> >>> >
> >>> > at javax.security.auth.Subject.doAs(Subject.java:415)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
> >>> >
> >>> > at
> >>> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743)
> >>> >
> >>> > at
> >>> >
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> >>> >
> >>> > at
> >>> >
> >>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>> >
> >>> > at
> >>> >
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>> >
> >>> > at java.lang.Thread.run(Thread.java:745)
> >>> >
> >>> > Caused by:
> >>> >
> >>>
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> >>> > GSS initiate failed
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
> >>> >
> >>> > at java.security.AccessController.doPrivileged(Native Method)
> >>> >
> >>> > at javax.security.auth.Subject.doAs(Subject.java:415)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >>> >
> >>> > at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
> >>> >
> >>> >
> >>> > On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yu...@gmail.com> wrote:
> >>> >>
> >>> >> bq. Attached both files
> >>> >>
> >>> >> I only found hbase-site.xml in the attachment.
> >>> >>
> >>> >> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <
> >>> manoj.murumkar@gmail.com>
> >>> >> wrote:
> >>> >>
> >>> >> > Attached both files.
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <
> >>> olorinbant@gmail.com>
> >>> >> > wrote:
> >>> >> >
> >>> >> >> Hi,
> >>> >> >>
> >>> >> >> Traces (especially one for region server) look a bit incomplete,
> did
> >>> >> >> you copy them fully?
> >>> >> >>
> >>> >> >> Also may help if you post relevant pieces of hbase-site.xml (with
> >>> >> >> security configs).
> >>> >> >>
> >>> >> >> Thanks,
> >>> >> >> Mikhail
> >>> >> >>
> >>> >> >>
> >>> >> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
> >>> >> >> <ma...@gmail.com> wrote:
> >>> >> >> > Hi,
> >>> >> >> >
> >>> >> >> > We have a secured cluster. All components are working well,
> except
> >>> >> >> hbase.
> >>> >> >> > Specifically, this is what I see on regionserver:
> >>> >> >> >
> >>> >> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
> >>> >> >> > ipc.RpcServer: Kerberos principal name is hbase/
> >>> >> >> > sfdvgctsn001.xxxxx@SFDVGCT.COM
> >>> >> >> >
> >>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >>> >> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> >>> >> >> >
> >>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >>> >> >> > ipc.RpcServer: Have read input token of size 627 for
> processing by
> >>> >> >> > saslServer.evaluateResponse()
> >>> >> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
> >>> >> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception
> >>> while
> >>> >> >> > reading:GSS initiate fail
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > and on the master:
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > java.io.IOException: Couldn't setup connection for
> >>> >> >> > hbase/sfdvgctmn004.
> >>> >> >> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
> >>> >> >> >
> >>> >> >> > at
> >>> >> >> >
> >>> >> >>
> >>> >> >>
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
> >>> >> >> >
> >>> >> >> > at java.security.AccessController.doPrivileged(Native
> >>> Method)
> >>> >> >> >
> >>> >> >> > at javax.security.auth.Subject.doAs(Subject.java:415)
> >>> >> >> >
> >>> >> >> > at
> >>> >> >> >
> >>> >> >>
> >>> >> >>
> >>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >>> >> >> >
> >>> >> >> > at
> >>> >> >> >
> >>> >> >>
> >>> >> >>
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
> >>> >> >> >
> >>> >> >> > at
> >>> >> >> >
> >>> >> >>
> >>> >> >>
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
> >>> >> >> >
> >>> >> >> > at
> >>> >> >> >
> >>> >> >>
> >>> >> >>
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
> >>> >> >> >
> >>> >> >> > at
> >>> >> >> >
> >>> >> >>
> >>> >> >>
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
> >>> >> >> >
> >>> >> >> > All the keytabs are set properly. Has anyone seen this before?
> >>> >> >> Appreciate
> >>> >> >> > the time.
> >>> >> >> >
> >>> >> >> > Manoj
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >> --
> >>> >> >> Thanks,
> >>> >> >> Michael Antonov
> >>> >> >>
> >>> >> >
> >>> >> >
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Thanks,
> >>> Michael Antonov
> >>>
> >>
> >>
>
>
>
> --
> Thanks,
> Michael Antonov
>
Re: Kerberos exception
Posted by Mikhail Antonov <ol...@gmail.com>.
Glad it helped!
In general though, in this cases, if you turn on java kerberos debug
logging (via something like export
HADOOP_OPTS="-Dsun.security.krb5.debug=true"), in your log you would
see error like: "unsupported key type found the default TGT: 18"
(which means encryption type 18 per kerberos specification, i.e.
aes256-cts-hmac-sha1-96, isn't supported by Java - due to lack of
unlimited strength JCE)
-Mikhail
On Mon, Mar 16, 2015 at 3:19 PM, Manoj Murumkar
<ma...@gmail.com> wrote:
> Thanks Mikhail! It turned out not having unlimited strength JCE files
> everywhere on the cluster was root cause (I reproduced it by leaving
> default files on regionservers on the newly built cluster).
>
> On Sun, Mar 15, 2015 at 9:48 PM, Manoj Murumkar <ma...@gmail.com>
> wrote:
>
>> Thanks. We took care of above items (all principals were generated using
>> Ambari tool) although I am not sure about 3rd item. One of my coworkers
>> decided to reinstall HBase, so I am waiting for the cluster to come up to
>> see if we still have the issue. Funny thing is, everything else worked in
>> secure environment (MR, Hive), just not HBase. The error message didn't
>> mention any details why the GSSException was raised (sample below):
>>
>>
>> 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
>> ipc.RpcServer: Kerberos principal name is hbase/
>> sfdvgctsn001.xxxxx@SFDVGCT.COM
>>
>> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>>
>> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> ipc.RpcServer: Have read input token of size 627 for processing by
>> saslServer.evaluateResponse()
>> *2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
>> ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
>> reading:GSS initiate fail*
>>
>>
>> On Sun, Mar 15, 2015 at 2:43 PM, Mikhail Antonov <ol...@gmail.com>
>> wrote:
>>
>>> I don't see region server log file in attachment for some reason.. May
>>> be the file is too big and gets rejected by mail server? Without it
>>> it'd be hard to say. Also master log looks a bit incomplete, are there
>>> no more traces? "GSS initiate fail" is normally followed by something
>>> like (just for example) - "[Caused by GSSException: No valid
>>> credentials provided (Mechanism level: Failed to find any Kerberos
>>> tgt)];" or so. Nothing like that?
>>>
>>> But in general, GSS initialization most often fails for following reasons:
>>>
>>> - wrong linux path or file permissions on keytab file
>>> - typo in kerberos principal/realm (that includes lower/upper case
>>> differences)
>>> - JCE unlimited strength file aren't installed
>>>
>>> On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar
>>> <ma...@gmail.com> wrote:
>>> > Attached region server log again.
>>> >
>>> > Ted,
>>> >
>>> > Corresponding log for master is:
>>> >
>>> > java.io.IOException: Couldn't setup connection for
>>> > hbase/sfdvgctmn001.gid.gap.com@SFDVGCT.COM to
>>> > hbase/sfdvgctsn001.gid.gap.com@S
>>> >
>>> > FDVGCT.COM
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>>> >
>>> > at java.security.AccessController.doPrivileged(Native Method)
>>> >
>>> > at javax.security.auth.Subject.doAs(Subject.java:415)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>>> >
>>> > at
>>> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743)
>>> >
>>> > at
>>> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>>> >
>>> > at
>>> >
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> >
>>> > at
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> >
>>> > at java.lang.Thread.run(Thread.java:745)
>>> >
>>> > Caused by:
>>> >
>>> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
>>> > GSS initiate failed
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
>>> >
>>> > at java.security.AccessController.doPrivileged(Native Method)
>>> >
>>> > at javax.security.auth.Subject.doAs(Subject.java:415)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>> >
>>> > at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
>>> >
>>> >
>>> > On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yu...@gmail.com> wrote:
>>> >>
>>> >> bq. Attached both files
>>> >>
>>> >> I only found hbase-site.xml in the attachment.
>>> >>
>>> >> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <
>>> manoj.murumkar@gmail.com>
>>> >> wrote:
>>> >>
>>> >> > Attached both files.
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <
>>> olorinbant@gmail.com>
>>> >> > wrote:
>>> >> >
>>> >> >> Hi,
>>> >> >>
>>> >> >> Traces (especially one for region server) look a bit incomplete, did
>>> >> >> you copy them fully?
>>> >> >>
>>> >> >> Also may help if you post relevant pieces of hbase-site.xml (with
>>> >> >> security configs).
>>> >> >>
>>> >> >> Thanks,
>>> >> >> Mikhail
>>> >> >>
>>> >> >>
>>> >> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
>>> >> >> <ma...@gmail.com> wrote:
>>> >> >> > Hi,
>>> >> >> >
>>> >> >> > We have a secured cluster. All components are working well, except
>>> >> >> hbase.
>>> >> >> > Specifically, this is what I see on regionserver:
>>> >> >> >
>>> >> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
>>> >> >> > ipc.RpcServer: Kerberos principal name is hbase/
>>> >> >> > sfdvgctsn001.xxxxx@SFDVGCT.COM
>>> >> >> >
>>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>>> >> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>>> >> >> >
>>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>>> >> >> > ipc.RpcServer: Have read input token of size 627 for processing by
>>> >> >> > saslServer.evaluateResponse()
>>> >> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
>>> >> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception
>>> while
>>> >> >> > reading:GSS initiate fail
>>> >> >> >
>>> >> >> >
>>> >> >> > and on the master:
>>> >> >> >
>>> >> >> >
>>> >> >> > java.io.IOException: Couldn't setup connection for
>>> >> >> > hbase/sfdvgctmn004.
>>> >> >> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
>>> >> >> >
>>> >> >> > at
>>> >> >> >
>>> >> >>
>>> >> >>
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>>> >> >> >
>>> >> >> > at java.security.AccessController.doPrivileged(Native
>>> Method)
>>> >> >> >
>>> >> >> > at javax.security.auth.Subject.doAs(Subject.java:415)
>>> >> >> >
>>> >> >> > at
>>> >> >> >
>>> >> >>
>>> >> >>
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>> >> >> >
>>> >> >> > at
>>> >> >> >
>>> >> >>
>>> >> >>
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>>> >> >> >
>>> >> >> > at
>>> >> >> >
>>> >> >>
>>> >> >>
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>>> >> >> >
>>> >> >> > at
>>> >> >> >
>>> >> >>
>>> >> >>
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>>> >> >> >
>>> >> >> > at
>>> >> >> >
>>> >> >>
>>> >> >>
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>>> >> >> >
>>> >> >> > All the keytabs are set properly. Has anyone seen this before?
>>> >> >> Appreciate
>>> >> >> > the time.
>>> >> >> >
>>> >> >> > Manoj
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> Thanks,
>>> >> >> Michael Antonov
>>> >> >>
>>> >> >
>>> >> >
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Michael Antonov
>>>
>>
>>
--
Thanks,
Michael Antonov
Re: Kerberos exception
Posted by Manoj Murumkar <ma...@gmail.com>.
Thanks Mikhail! It turned out not having unlimited strength JCE files
everywhere on the cluster was root cause (I reproduced it by leaving
default files on regionservers on the newly built cluster).
On Sun, Mar 15, 2015 at 9:48 PM, Manoj Murumkar <ma...@gmail.com>
wrote:
> Thanks. We took care of above items (all principals were generated using
> Ambari tool) although I am not sure about 3rd item. One of my coworkers
> decided to reinstall HBase, so I am waiting for the cluster to come up to
> see if we still have the issue. Funny thing is, everything else worked in
> secure environment (MR, Hive), just not HBase. The error message didn't
> mention any details why the GSSException was raised (sample below):
>
>
> 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
> ipc.RpcServer: Kerberos principal name is hbase/
> sfdvgctsn001.xxxxx@SFDVGCT.COM
>
> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>
> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> ipc.RpcServer: Have read input token of size 627 for processing by
> saslServer.evaluateResponse()
> *2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
> ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
> reading:GSS initiate fail*
>
>
> On Sun, Mar 15, 2015 at 2:43 PM, Mikhail Antonov <ol...@gmail.com>
> wrote:
>
>> I don't see region server log file in attachment for some reason.. May
>> be the file is too big and gets rejected by mail server? Without it
>> it'd be hard to say. Also master log looks a bit incomplete, are there
>> no more traces? "GSS initiate fail" is normally followed by something
>> like (just for example) - "[Caused by GSSException: No valid
>> credentials provided (Mechanism level: Failed to find any Kerberos
>> tgt)];" or so. Nothing like that?
>>
>> But in general, GSS initialization most often fails for following reasons:
>>
>> - wrong linux path or file permissions on keytab file
>> - typo in kerberos principal/realm (that includes lower/upper case
>> differences)
>> - JCE unlimited strength file aren't installed
>>
>> On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar
>> <ma...@gmail.com> wrote:
>> > Attached region server log again.
>> >
>> > Ted,
>> >
>> > Corresponding log for master is:
>> >
>> > java.io.IOException: Couldn't setup connection for
>> > hbase/sfdvgctmn001.gid.gap.com@SFDVGCT.COM to
>> > hbase/sfdvgctsn001.gid.gap.com@S
>> >
>> > FDVGCT.COM
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>> >
>> > at java.security.AccessController.doPrivileged(Native Method)
>> >
>> > at javax.security.auth.Subject.doAs(Subject.java:415)
>> >
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>> >
>> > at
>> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743)
>> >
>> > at
>> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>> >
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > Caused by:
>> >
>> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
>> > GSS initiate failed
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
>> >
>> > at java.security.AccessController.doPrivileged(Native Method)
>> >
>> > at javax.security.auth.Subject.doAs(Subject.java:415)
>> >
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
>> >
>> >
>> > On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yu...@gmail.com> wrote:
>> >>
>> >> bq. Attached both files
>> >>
>> >> I only found hbase-site.xml in the attachment.
>> >>
>> >> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <
>> manoj.murumkar@gmail.com>
>> >> wrote:
>> >>
>> >> > Attached both files.
>> >> >
>> >> >
>> >> >
>> >> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <
>> olorinbant@gmail.com>
>> >> > wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> Traces (especially one for region server) look a bit incomplete, did
>> >> >> you copy them fully?
>> >> >>
>> >> >> Also may help if you post relevant pieces of hbase-site.xml (with
>> >> >> security configs).
>> >> >>
>> >> >> Thanks,
>> >> >> Mikhail
>> >> >>
>> >> >>
>> >> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
>> >> >> <ma...@gmail.com> wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> > We have a secured cluster. All components are working well, except
>> >> >> hbase.
>> >> >> > Specifically, this is what I see on regionserver:
>> >> >> >
>> >> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
>> >> >> > ipc.RpcServer: Kerberos principal name is hbase/
>> >> >> > sfdvgctsn001.xxxxx@SFDVGCT.COM
>> >> >> >
>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> >> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>> >> >> >
>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> >> >> > ipc.RpcServer: Have read input token of size 627 for processing by
>> >> >> > saslServer.evaluateResponse()
>> >> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
>> >> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception
>> while
>> >> >> > reading:GSS initiate fail
>> >> >> >
>> >> >> >
>> >> >> > and on the master:
>> >> >> >
>> >> >> >
>> >> >> > java.io.IOException: Couldn't setup connection for
>> >> >> > hbase/sfdvgctmn004.
>> >> >> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
>> >> >> >
>> >> >> > at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>> >> >> >
>> >> >> > at java.security.AccessController.doPrivileged(Native
>> Method)
>> >> >> >
>> >> >> > at javax.security.auth.Subject.doAs(Subject.java:415)
>> >> >> >
>> >> >> > at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> >> >> >
>> >> >> > at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>> >> >> >
>> >> >> > at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>> >> >> >
>> >> >> > at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>> >> >> >
>> >> >> > at
>> >> >> >
>> >> >>
>> >> >>
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>> >> >> >
>> >> >> > All the keytabs are set properly. Has anyone seen this before?
>> >> >> Appreciate
>> >> >> > the time.
>> >> >> >
>> >> >> > Manoj
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Thanks,
>> >> >> Michael Antonov
>> >> >>
>> >> >
>> >> >
>> >
>> >
>>
>>
>>
>> --
>> Thanks,
>> Michael Antonov
>>
>
>
Re: Kerberos exception
Posted by Manoj Murumkar <ma...@gmail.com>.
Thanks. We took care of above items (all principals were generated using
Ambari tool) although I am not sure about 3rd item. One of my coworkers
decided to reinstall HBase, so I am waiting for the cluster to come up to
see if we still have the issue. Funny thing is, everything else worked in
secure environment (MR, Hive), just not HBase. The error message didn't
mention any details why the GSSException was raised (sample below):
2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
ipc.RpcServer: Kerberos principal name is hbase/
sfdvgctsn001.xxxxx@SFDVGCT.COM
2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
ipc.RpcServer: Created SASL server with mechanism = GSSAPI
2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
ipc.RpcServer: Have read input token of size 627 for processing by
saslServer.evaluateResponse()
*2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
reading:GSS initiate fail*
On Sun, Mar 15, 2015 at 2:43 PM, Mikhail Antonov <ol...@gmail.com>
wrote:
> I don't see region server log file in attachment for some reason.. May
> be the file is too big and gets rejected by mail server? Without it
> it'd be hard to say. Also master log looks a bit incomplete, are there
> no more traces? "GSS initiate fail" is normally followed by something
> like (just for example) - "[Caused by GSSException: No valid
> credentials provided (Mechanism level: Failed to find any Kerberos
> tgt)];" or so. Nothing like that?
>
> But in general, GSS initialization most often fails for following reasons:
>
> - wrong linux path or file permissions on keytab file
> - typo in kerberos principal/realm (that includes lower/upper case
> differences)
> - JCE unlimited strength file aren't installed
>
> On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar
> <ma...@gmail.com> wrote:
> > Attached region server log again.
> >
> > Ted,
> >
> > Corresponding log for master is:
> >
> > java.io.IOException: Couldn't setup connection for
> > hbase/sfdvgctmn001.gid.gap.com@SFDVGCT.COM to
> > hbase/sfdvgctsn001.gid.gap.com@S
> >
> > FDVGCT.COM
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
> >
> > at java.security.AccessController.doPrivileged(Native Method)
> >
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> >
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
> >
> > at
> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
> >
> > at
> >
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176)
> >
> > at
> >
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670)
> >
> > at
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004)
> >
> > at
> >
> org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119)
> >
> > at
> >
> org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743)
> >
> > at
> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> >
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Caused by:
> > org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> > GSS initiate failed
> >
> > at
> >
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)
> >
> > at
> >
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
> >
> > at java.security.AccessController.doPrivileged(Native Method)
> >
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> >
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
> >
> >
> > On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> bq. Attached both files
> >>
> >> I only found hbase-site.xml in the attachment.
> >>
> >> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <
> manoj.murumkar@gmail.com>
> >> wrote:
> >>
> >> > Attached both files.
> >> >
> >> >
> >> >
> >> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <
> olorinbant@gmail.com>
> >> > wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> Traces (especially one for region server) look a bit incomplete, did
> >> >> you copy them fully?
> >> >>
> >> >> Also may help if you post relevant pieces of hbase-site.xml (with
> >> >> security configs).
> >> >>
> >> >> Thanks,
> >> >> Mikhail
> >> >>
> >> >>
> >> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
> >> >> <ma...@gmail.com> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > We have a secured cluster. All components are working well, except
> >> >> hbase.
> >> >> > Specifically, this is what I see on regionserver:
> >> >> >
> >> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
> >> >> > ipc.RpcServer: Kerberos principal name is hbase/
> >> >> > sfdvgctsn001.xxxxx@SFDVGCT.COM
> >> >> >
> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> >> >> >
> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >> >> > ipc.RpcServer: Have read input token of size 627 for processing by
> >> >> > saslServer.evaluateResponse()
> >> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
> >> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception
> while
> >> >> > reading:GSS initiate fail
> >> >> >
> >> >> >
> >> >> > and on the master:
> >> >> >
> >> >> >
> >> >> > java.io.IOException: Couldn't setup connection for
> >> >> > hbase/sfdvgctmn004.
> >> >> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
> >> >> >
> >> >> > at
> >> >> >
> >> >>
> >> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
> >> >> >
> >> >> > at java.security.AccessController.doPrivileged(Native
> Method)
> >> >> >
> >> >> > at javax.security.auth.Subject.doAs(Subject.java:415)
> >> >> >
> >> >> > at
> >> >> >
> >> >>
> >> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >> >> >
> >> >> > at
> >> >> >
> >> >>
> >> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
> >> >> >
> >> >> > at
> >> >> >
> >> >>
> >> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
> >> >> >
> >> >> > at
> >> >> >
> >> >>
> >> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
> >> >> >
> >> >> > at
> >> >> >
> >> >>
> >> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
> >> >> >
> >> >> > All the keytabs are set properly. Has anyone seen this before?
> >> >> Appreciate
> >> >> > the time.
> >> >> >
> >> >> > Manoj
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Thanks,
> >> >> Michael Antonov
> >> >>
> >> >
> >> >
> >
> >
>
>
>
> --
> Thanks,
> Michael Antonov
>
Re: Kerberos exception
Posted by Mikhail Antonov <ol...@gmail.com>.
I don't see region server log file in attachment for some reason.. May
be the file is too big and gets rejected by mail server? Without it
it'd be hard to say. Also master log looks a bit incomplete, are there
no more traces? "GSS initiate fail" is normally followed by something
like (just for example) - "[Caused by GSSException: No valid
credentials provided (Mechanism level: Failed to find any Kerberos
tgt)];" or so. Nothing like that?
But in general, GSS initialization most often fails for following reasons:
- wrong linux path or file permissions on keytab file
- typo in kerberos principal/realm (that includes lower/upper case differences)
- JCE unlimited strength file aren't installed
On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar
<ma...@gmail.com> wrote:
> Attached region server log again.
>
> Ted,
>
> Corresponding log for master is:
>
> java.io.IOException: Couldn't setup connection for
> hbase/sfdvgctmn001.gid.gap.com@SFDVGCT.COM to
> hbase/sfdvgctsn001.gid.gap.com@S
>
> FDVGCT.COM
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
>
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176)
>
> at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119)
>
> at
> org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743)
>
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by:
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> GSS initiate failed
>
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)
>
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
>
>
> On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>> bq. Attached both files
>>
>> I only found hbase-site.xml in the attachment.
>>
>> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <ma...@gmail.com>
>> wrote:
>>
>> > Attached both files.
>> >
>> >
>> >
>> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <ol...@gmail.com>
>> > wrote:
>> >
>> >> Hi,
>> >>
>> >> Traces (especially one for region server) look a bit incomplete, did
>> >> you copy them fully?
>> >>
>> >> Also may help if you post relevant pieces of hbase-site.xml (with
>> >> security configs).
>> >>
>> >> Thanks,
>> >> Mikhail
>> >>
>> >>
>> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
>> >> <ma...@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > We have a secured cluster. All components are working well, except
>> >> hbase.
>> >> > Specifically, this is what I see on regionserver:
>> >> >
>> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
>> >> > ipc.RpcServer: Kerberos principal name is hbase/
>> >> > sfdvgctsn001.xxxxx@SFDVGCT.COM
>> >> >
>> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>> >> >
>> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> >> > ipc.RpcServer: Have read input token of size 627 for processing by
>> >> > saslServer.evaluateResponse()
>> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
>> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
>> >> > reading:GSS initiate fail
>> >> >
>> >> >
>> >> > and on the master:
>> >> >
>> >> >
>> >> > java.io.IOException: Couldn't setup connection for
>> >> > hbase/sfdvgctmn004.
>> >> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
>> >> >
>> >> > at
>> >> >
>> >>
>> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>> >> >
>> >> > at java.security.AccessController.doPrivileged(Native Method)
>> >> >
>> >> > at javax.security.auth.Subject.doAs(Subject.java:415)
>> >> >
>> >> > at
>> >> >
>> >>
>> >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> >> >
>> >> > at
>> >> >
>> >>
>> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>> >> >
>> >> > at
>> >> >
>> >>
>> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>> >> >
>> >> > at
>> >> >
>> >>
>> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>> >> >
>> >> > at
>> >> >
>> >>
>> >> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>> >> >
>> >> > All the keytabs are set properly. Has anyone seen this before?
>> >> Appreciate
>> >> > the time.
>> >> >
>> >> > Manoj
>> >>
>> >>
>> >>
>> >> --
>> >> Thanks,
>> >> Michael Antonov
>> >>
>> >
>> >
>
>
--
Thanks,
Michael Antonov
Re: Kerberos exception
Posted by Manoj Murumkar <ma...@gmail.com>.
Attached region server log again.
Ted,
Corresponding log for master is:
java.io.IOException: Couldn't setup connection for hbase/
sfdvgctmn001.gid.gap.com@SFDVGCT.COM to hbase/sfdvgctsn001.gid.gap.com@S
FDVGCT.COM
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004)
at
org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119)
at
org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
GSS initiate failed
at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)
at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yu...@gmail.com> wrote:
> bq. Attached both files
>
> I only found hbase-site.xml in the attachment.
>
> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <ma...@gmail.com>
> wrote:
>
> > Attached both files.
> >
> >
> >
> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <ol...@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> Traces (especially one for region server) look a bit incomplete, did
> >> you copy them fully?
> >>
> >> Also may help if you post relevant pieces of hbase-site.xml (with
> >> security configs).
> >>
> >> Thanks,
> >> Mikhail
> >>
> >>
> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
> >> <ma...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > We have a secured cluster. All components are working well, except
> >> hbase.
> >> > Specifically, this is what I see on regionserver:
> >> >
> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
> >> > ipc.RpcServer: Kerberos principal name is hbase/
> >> > sfdvgctsn001.xxxxx@SFDVGCT.COM
> >> >
> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> >> >
> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >> > ipc.RpcServer: Have read input token of size 627 for processing by
> >> > saslServer.evaluateResponse()
> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
> >> > reading:GSS initiate fail
> >> >
> >> >
> >> > and on the master:
> >> >
> >> >
> >> > java.io.IOException: Couldn't setup connection for hbase/sfdvgctmn004.
> >> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
> >> >
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
> >> >
> >> > at java.security.AccessController.doPrivileged(Native Method)
> >> >
> >> > at javax.security.auth.Subject.doAs(Subject.java:415)
> >> >
> >> > at
> >> >
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >> >
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
> >> >
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
> >> >
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
> >> >
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
> >> >
> >> > All the keytabs are set properly. Has anyone seen this before?
> >> Appreciate
> >> > the time.
> >> >
> >> > Manoj
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Michael Antonov
> >>
> >
> >
>
Re: Kerberos exception
Posted by Ted Yu <yu...@gmail.com>.
bq. Attached both files
I only found hbase-site.xml in the attachment.
On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <ma...@gmail.com>
wrote:
> Attached both files.
>
>
>
> On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <ol...@gmail.com>
> wrote:
>
>> Hi,
>>
>> Traces (especially one for region server) look a bit incomplete, did
>> you copy them fully?
>>
>> Also may help if you post relevant pieces of hbase-site.xml (with
>> security configs).
>>
>> Thanks,
>> Mikhail
>>
>>
>> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
>> <ma...@gmail.com> wrote:
>> > Hi,
>> >
>> > We have a secured cluster. All components are working well, except
>> hbase.
>> > Specifically, this is what I see on regionserver:
>> >
>> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
>> > ipc.RpcServer: Kerberos principal name is hbase/
>> > sfdvgctsn001.xxxxx@SFDVGCT.COM
>> >
>> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>> >
>> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> > ipc.RpcServer: Have read input token of size 627 for processing by
>> > saslServer.evaluateResponse()
>> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
>> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
>> > reading:GSS initiate fail
>> >
>> >
>> > and on the master:
>> >
>> >
>> > java.io.IOException: Couldn't setup connection for hbase/sfdvgctmn004.
>> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>> >
>> > at java.security.AccessController.doPrivileged(Native Method)
>> >
>> > at javax.security.auth.Subject.doAs(Subject.java:415)
>> >
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>> >
>> > All the keytabs are set properly. Has anyone seen this before?
>> Appreciate
>> > the time.
>> >
>> > Manoj
>>
>>
>>
>> --
>> Thanks,
>> Michael Antonov
>>
>
>
Re: Kerberos exception
Posted by Manoj Murumkar <ma...@gmail.com>.
Attached both files.
On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <ol...@gmail.com>
wrote:
> Hi,
>
> Traces (especially one for region server) look a bit incomplete, did
> you copy them fully?
>
> Also may help if you post relevant pieces of hbase-site.xml (with
> security configs).
>
> Thanks,
> Mikhail
>
> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
> <ma...@gmail.com> wrote:
> > Hi,
> >
> > We have a secured cluster. All components are working well, except hbase.
> > Specifically, this is what I see on regionserver:
> >
> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
> > ipc.RpcServer: Kerberos principal name is hbase/
> > sfdvgctsn001.xxxxx@SFDVGCT.COM
> >
> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> >
> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> > ipc.RpcServer: Have read input token of size 627 for processing by
> > saslServer.evaluateResponse()
> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
> > reading:GSS initiate fail
> >
> >
> > and on the master:
> >
> >
> > java.io.IOException: Couldn't setup connection for hbase/sfdvgctmn004.
> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
> >
> > at java.security.AccessController.doPrivileged(Native Method)
> >
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> >
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
> >
> > at
> >
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
> >
> > All the keytabs are set properly. Has anyone seen this before? Appreciate
> > the time.
> >
> > Manoj
>
>
>
> --
> Thanks,
> Michael Antonov
>
Re: Kerberos exception
Posted by Mikhail Antonov <ol...@gmail.com>.
Hi,
Traces (especially one for region server) look a bit incomplete, did
you copy them fully?
Also may help if you post relevant pieces of hbase-site.xml (with
security configs).
Thanks,
Mikhail
On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
<ma...@gmail.com> wrote:
> Hi,
>
> We have a secured cluster. All components are working well, except hbase.
> Specifically, this is what I see on regionserver:
>
> 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
> ipc.RpcServer: Kerberos principal name is hbase/
> sfdvgctsn001.xxxxx@SFDVGCT.COM
>
> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>
> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> ipc.RpcServer: Have read input token of size 627 for processing by
> saslServer.evaluateResponse()
> 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
> ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
> reading:GSS initiate fail
>
>
> and on the master:
>
>
> java.io.IOException: Couldn't setup connection for hbase/sfdvgctmn004.
> xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>
> All the keytabs are set properly. Has anyone seen this before? Appreciate
> the time.
>
> Manoj
--
Thanks,
Michael Antonov