HBase 2.1 client intermittent Kerberos failure in IBM JVM

[K Yip <ky...@yahoo.com.INVALID>]

Hi,
I have an application  that has been working with HBase 1.x servers using Kerberos authentication for a while. 

I upgraded the application to support HBase 2.1 servers recently. The application is working fine in Oracle JVM but not in IBM JVM (both Java 1.8). 

In IBM JVM, after the successful UserGroupInformation.loginUserFromKeytab(), it always fails to find the javax.security.auth.Subject during the PROCESS_TGS step and the TGS_REQ was never sent for the /hbase service. So, in order to address this, I made use of UserGroupInformation.getCurrentUser().doAs(<my HBase operation>) where <my HBase operation> can be HBase available check, connection creation, get table names, table scan, put, get, etc. This approach seems to work except I am facing intermittent failures where the following error is logged:
[2/7/20 6:50:20:682 GMT] 0000014e SystemErr                                                    R javax.security.sasl.SaslException: Call to eng-bigbang-hadoop01.rpega.com/10.20.204.19:16020 failed on local exception: javax.security.sasl.SaslException: Failure to initialize security context [Caused by org.ietf.jgss.GSSException, major code: 11, minor code: 0
        major string: General failure, unspecified at GSSAPI level
        minor string: Cannot get credential for principal default principal] [Caused by javax.security.sasl.SaslException: Failure to initialize security context [Caused by org.ietf.jgss.GSSException, major code: 11, minor code: 0
        major string: General failure, unspecified at GSSAPI level
        minor string: Cannot get credential for principal default principal]]

This is the same error that consistently happens before I used the UserGroupInformation.getCurrentUser().doAs(<my HBase operation>) technique. It seems to me somehow the "login context" was lost occasionally and that is why the logged in Subject cannot be found. 

Not sure how this is relevant to the issue here. From my debugging sessions, I notice is that HBase 1.x performs the PROCESS_TGS step in the same thread as the initial steps while HBase 2.1 performs the step in a separate thread. 

Since my application has been working with HBase 1.x servers (in both Oracle and IBM JVM's) and my application also works properly with HDFS services in Kerberos configuration in both Oracle and IBM JVM's, this seems to be a HBase 2.x issue. (I also tried HBase 2.2 client jars which did not help.)

Any suggestion on how to address or troubleshoot this issue is greatly appreciated. 


Best Regards,
Kai