HBASE-5780

[Tony Dean <To...@sas.com>]

Hi,

I upgraded to hbase 0.94.0 which was supposed to fix HBASE-5780 issue.  It initially worked but I am still getting intermittent failures when region servers communicate with zookeeper.  It appears that the authentication is successful, but 

On the zookeeper server:
12/07/17 16:47:16 INFO auth.SaslServerCallbackHandler: Successfully authenticated client: authenticationID=hbase/rdcesx12082.race.sas.com@OBSIDIAN.SAS.COM;  authorizationID=hbase/rdcesx12082.race.sas.com@OBSIDIAN.SAS.COM.
12/07/17 16:47:16 INFO auth.SaslServerCallbackHandler: Setting authorizedID: hbase
12/07/17 16:47:16 INFO server.ZooKeeperServer: adding SASL authorization for authorizationID: hbase
12/07/17 16:47:21 WARN server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1389637a5f10003, likely client has closed socket

Looks like the client hbase/rdcesx12082.race.sas.com@OBSIDIAN.SAS.COM authenticated successfully.  Then it received an unexpected end of stream.

On the server rdcesx11017.race.sas.com, I see the following:
12/07/17 16:47:16 INFO zookeeper.ClientCnxn: Session establishment complete on server rdcesx11017.race.sas.com/10.21.19.11:2181, sessionid = 0x1389637a5f10003, negotiated timeout = 180000
12/07/17 16:47:16 INFO zookeeper.Login: TGT valid starting at:        Tue Jul 17 16:47:01 EDT 2012
12/07/17 16:47:16 INFO zookeeper.Login: TGT expires:                  Wed Jul 18 16:47:01 EDT 2012
12/07/17 16:47:16 INFO zookeeper.Login: TGT refresh sleeping until: Wed Jul 18 13:09:32 EDT 2012
12/07/17 16:47:16 INFO regionserver.ShutdownHook: Installed shutdown hook thread: Shutdownhook:regionserver60020
                No encryption was performed by peer.
12/07/17 16:47:16 WARN zookeeper.ZKUtil: regionserver:60020-0x1389637a5f10003 Unable to get data of znode /hbase/shutdown
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /hbase/shutdown

It appears that the session was established, then "Unable to get data of znode /hbase/shutdown".

Previously, I killed the process so maybe the znode data was not saved appropriately.  Is there a way to make sure zookeeper data is clean?

Thanks.