Kerberized HiveServer2 Parquet Join exception

[Dong Bo 董博 <Do...@lavapm.com>]

Hello,

I try to run a join query on HiveServer2 with Kerberos , return error from it , both tables are in parquet format , but test queries on text tables are working fine .

INFO  : Compiling command(queryId=hive_20180716143131_0657a760-2272-41f3-aea4-036082ec1d82): select * from  dim_city_district a join dim_city_level b
on a.id = b.city_id
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:a.id, type:string, comment:null), FieldSchema(name:a.provience_name, type:string, comment:null), FieldSchema(name:a.provience_code, type:string, comment:null), FieldSchema(name:a.city_name, type:string, comment:null), FieldSchema(name:a.city_code, type:string, comment:null), FieldSchema(name:a.district_name, type:string, comment:null), FieldSchema(name:a.district_code, type:string, comment:null), FieldSchema(name:a.district_postcode, type:string, comment:null), FieldSchema(name:b.city_id, type:int, comment:null), FieldSchema(name:b.city_name, type:string, comment:null), FieldSchema(name:b.city_level, type:int, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20180716143131_0657a760-2272-41f3-aea4-036082ec1d82); Time taken: 0.161 seconds
INFO  : Executing command(queryId=hive_20180716143131_0657a760-2272-41f3-aea4-036082ec1d82): select * from  dim_city_district a join dim_city_level b
on a.id = b.city_id
INFO  : Query ID = hive_20180716143131_0657a760-2272-41f3-aea4-036082ec1d82
INFO  : Total jobs = 1
INFO  : Starting task [Stage-4:MAPREDLOCAL] in serial mode
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
18/07/16 14:31:07 WARN conf.HiveConf: HiveConf of name hive.server2.idle.session.timeout_check_operation does not exist
18/07/16 14:31:07 WARN conf.HiveConf: HiveConf of name hive.sentry.conf.url does not exist
18/07/16 14:31:07 WARN conf.HiveConf: HiveConf of name hive.server2.authorization.external.exec does not exist
18/07/16 14:31:07 WARN conf.HiveConf: HiveConf of name hive.entity.capture.input.URI does not exist
Execution log at: /tmp/hive/hive_20180716143131_0657a760-2272-41f3-aea4-036082ec1d82.log
2018-07-16 02:31:07 Starting to launch local task to process map join;          maximum memory = 518979584
ERROR : Execution failed with exit status: 2
ERROR : Obtaining error information
ERROR :
Task failed!
Task ID:
  Stage-4

Logs:

ERROR : /mnt/data1/log/hive/hadoop-cmf-hive-HIVESERVER2-cdh-master.lavapm.log.out
ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
INFO  : Completed executing command(queryId=hive_20180716143131_0657a760-2272-41f3-aea4-036082ec1d82); Time taken: 4.889 seconds


Checked in the Execution Log , found Kerberos Error in the log file :


2018-07-16 14:15:10,711 INFO  [main]: mr.MapredLocalTask (MapredLocalTask.java:initializeOperators(502)) - fetchoperator for b initialized
2018-07-16 14:15:11,266 WARN  [main]: security.UserGroupInformation (UserGroupInformation.java:doAs(1920)) - PriviledgedActionException as:hive (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2018-07-16 14:15:11,267 WARN  [main]: ipc.Client (Client.java:run(713)) - Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2018-07-16 14:15:11,268 WARN  [main]: security.UserGroupInformation (UserGroupInformation.java:doAs(1920)) - PriviledgedActionException as:hive (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
2018-07-16 14:15:11,272 ERROR [main]: mr.MapredLocalTask (MapredLocalTask.java:executeInProcess(398)) - Hive Runtime Error: Map local work failed
java.io.IOException: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "cdh-master.lavapm/192.168.0.91"; destination host is: "cdh-master.lavapm":8020;
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:446)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:421)
        at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:387)
        at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:766)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "cdh-master.lavapm/192.168.0.91"; destination host is: "cdh-master.lavapm":8020;


But , other task stages（like task stage 3）can be executed and return result , also HiveServer2 is configured with a working keytab file , could someone help to know where the hiveserver2 is going to get a keytab file for task stage 4?
Or any settings I should check to fix this ?

BTW, set hive.auto.convert.join = false could skip this problem , but this is not a fix.

Thanks

Carl