oozie java action, failed using keytab to connect to kerberized hive server 2

[Jary Du <ja...@gmail.com>]

I am having a oozie java action and it tries to connect to kerberized Hive Server 2 with the keytab.
Using keytab to login into kerberos was successful by doing “UserGroupInformation.loginUserFromKeytab(principalName, KEYTAB);”
But accessing hive server2 stills complains failed to find any kerberos tgt. 

I used the same code on my app node which was successful, but when running the same thing in the oozie java action, it fails with exception:
(the only difference I can think of was the "code running on app node" vs "the code running on data node”)

Does anyone know why it was like this?

2015-08-27 21:17:46,839 ERROR [main] org.apache.thrift.transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
	at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
	at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
	at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:190)
	at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:163)
	at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
	at java.sql.DriverManager.getConnection(DriverManager.java:579)
	at java.sql.DriverManager.getConnection(DriverManager.java:190)
	***
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
	... 29 more
2015-08-27 21:17:46,843 INFO [main
My oozie java action main
   public static void main(String[] args) throws Exception {
	
	***
        
	UserGroupInformation.loginUserFromKeytab(principalName, KEYTAB);
        logger.info("Logged into Kerberos with principal[{}] and keytab[{}]", PRINCIPAL, KEYTAB);

        try (Connection conn = HiveConnectionBroker.getConnection(); Statement stmt = conn.createStatement()) {
            ResultSet rs = stmt.executeQuery("SHOW DATABASES LIKE ‘db'");
            if (rs.next())
                System.out.println("SUCCESS!!!!!!!!!!!!!!!!!!");
            else
                System.out.println("FAILED!!!!!!!!!!!!!!!!!");
        } catch (Exception e) {
            System.out.print(e);
        }
}