Problem with Scan operation over a Secure HBase 1.0.0-CDH5.4

[Gabriela Montiel Moreno <ga...@oracle.com>]

Hello,

We are running a Secure HBase cluster (enabling kerberos authentication and setting up hbase authorization) and we are trying to execute operations using a Java client. We are using the following configuration. 

import org.apache.hadoop.security.*;
szQuorum="node01.example.com,node02.example.com,node01.example.com";

config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", szQuorum);
config.set("hbase.zookeeper.property.clientPort", "2181");
config.set("hbase.security.authentication", "kerberos");
config.set("hadoop.security.authentication", "kerberos");
config.set("hbase.master.kerberos.principal","hbase/node03.example.com@EXAMPLE.COM");
config.set("hbase.regionserver.kerberos.principal","hbase/node03.example.com@EXAMPLE.COM");

UserGroupInformation.setConfiguration(config);
ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI("hbase/node03.example.com@EXAMPLE.COM","/var/run/cloudera-scm-agent/process/224-hbase-REGIONSERVER/hbase.keytab"); 
UserGroupInformation.setLoginUser(ugi);

hconn = HConnectionManager.createConnection(config);
hti=conn.getTable("exampletbl");

scan = new Scan();
rsScanner=hti.getScanner(scan); 

While we are able to create a table, puts and gets, when we try to execute a scan after a few seconds we get the following exceptions:

97976 [hconnection-0x4f2c9ba6-shared--pool1-t6] DEBUG org.apache.hadoop.hbase.security.HBaseSaslRpcClient  - Have sent token of size 674 from initSASLContext.
97977 [hconnection-0x4f2c9ba6-shared--pool1-t6] WARN  org.apache.hadoop.security.UserGroupInformation  - PriviledgedActionExceptionas:hbase/node01.example.com@EXAMPLE.COM(auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
97977 [hconnection-0x4f2c9ba6-shared--pool1-t6] DEBUG org.apache.hadoop.security.UserGroupInformation  - PrivilegedAction as:hbase/node01.example.com@EXAMPLE.COM(auth:KERBEROS) from:org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:631)
97977 [hconnection-0x4f2c9ba6-shared--pool1-t6] WARN  org.apache.hadoop.hbase.ipc.AbstractRpcClient  - Couldn't setup connection for hbase/node01.example.com@EXAMPLE.COM tohbase/node01.example.com@EXAMPLE.COM

We have run the kinit and setup the jaas.conf in the JAVA_OPTIONS of our Java application.  

export JAVA_OPTIONS=" -Djava.security.auth.login.config=/var/run/cloudera-scm-agent/process/224-hbase-REGIONSERVER/jaas.conf -Dsun.security.krb5.debug=true "

kinit -k -t /var/run/cloudera-scm-agent/process/224-hbase-REGIONSERVER/hbase.keytabhbase/hbase/node03.example.com@EXAMPLE.COM

klist -f
Ticket cache: FILE:/tmp/krb5cc_0
Default principal:hbase/hbase/node03.example.com@EXAMPLE.COM

Valid starting     Expires            Service principal
06/17/15 17:37:31  06/18/15 17:37:31 krbtgt/EXAMPLE.COM@EXAMPLE.COM
    renew until 06/22/15 17:37:31, Flags: FRI

less /var/run/cloudera-scm-agent/process/224-hbase-REGIONSERVER/jaas.conf

Client {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  useTicketCache=true
  keyTab="hbase.keytab"
  principal="hbase/hbase/node03.example.com@EXAMPLE.COM";
};

Is there any missing configuration?

Thanks,

Gaby