issues authenticating on hdfs sink using kerberos

[yoandy terradas <kw...@hotmail.com>]

hi, 

im using flume 1.6.0 agent setup kafka-source, memory-channel and hdfs-sink.  the hdfs has kerberos setup with proxy users.  i cannot get the sink to write because it fails saying i didnt provide kerberos principal.  

2016-07-13 17:25:41,738 (conf-file-poller-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-07-13 17:25:41,853 (conf-file-poller-0) [INFO - org.apache.flume.auth.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:172)] Attempting kerberos login as principal (ebee/01-ebee-autoenv.envnxs.net@REALM.COM) from keytab file (/home/yt/flume/conf/krb5.keytab.ebee)
2016-07-13 17:25:42,252 (conf-file-poller-0) [INFO - org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:979)] Login successful for user ebee/01-ebee-autoenv.envnxs.net@REALM.COM using keytab file /home/yt/flume/conf/krb5.keytab.ebee
2016-07-13 17:25:42,253 (conf-file-poller-0) [INFO - org.apache.flume.auth.KerberosAuthenticator.printUGI(KerberosAuthenticator.java:191)] 
Logged as:  
User: ebee/01-ebee-autoenv.envnxs.net@REALM.COM
Auth method: KERBEROS 
Keytab: true 

2016-07-13 17:25:42,254 (conf-file-poller-0) [INFO - org.apache.flume.auth.KerberosAuthenticator.printUGI(KerberosAuthenticator.java:191)] 
Proxy as:  
User: pub_user 
Auth method: PROXY 
Keytab: false 

2016-07-13 17:25:42,257 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:114)] Channel mem-ch1 connected to [kafka-sc1, hdfs-sk1]
 
that looks like authentication was successful.  it connects to kafka and start putting messages in the channel, then 

2016-07-13 17:25:44,304 (ConsumerFetcherThread-flume_01.local-hostname-1468430742665-7ece6605-0-80) [INFO - kafka.utils.Logging$class.info(Logging.scala:68)] [ConsumerFetcherThread-flume_01.rufus.sand-08.lax1-1468430742665-7ece6605-0-80], Starting 
2016-07-13 17:25:44,643 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.HDFSSequenceFile.configure(HDFSSequenceFile.java:63)] writeFormat = Text, UseRawLocalFileSystem = false
2016-07-13 17:25:44,695 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:234)] Creating hdfs://01-hadoop-namenode-hostname/pubs/logs/opt_rtpp_mapper_log_record_joined_pb/16-07-13-17-25/events.1468430744636.tmp
2016-07-13 17:25:45,918 (hdfs-hdfs-sk1-call-runner-0) [WARN - org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:675)] Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
2016-07-13 17:25:45,919 (hdfs-hdfs-sk1-call-runner-0) [WARN - org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1674)] PriviledgedActionException as:ebee/01-ebee-autoenv.envnxs.net@REALM.COM (auth:KERBEROS) cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
2016-07-13 17:25:45,933 (hdfs-hdfs-sk1-call-runner-0) [WARN - org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1674)] PriviledgedActionException as:dpaas_publishers_user (auth:PROXY) via ebee/01-ebee-autoenv.envnxs.net@REAL.COM (auth:KERBEROS) cause:java.io.IOException: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "01.local-hostname/local-ip-address"; destination host is: "01-hadoop-namenode-hostname":8020; 
2016-07-13 17:25:45,935 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed
org.apache.flume.auth.SecurityException: Privileged action failed
at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:49)
at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "01.local-hostname/local-ip-address"; destination host is: "01-hadoop-namenode-hostname":8020; 
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)

looking at the code for hdfs-sink it seems as the kerberos auth is propagated.  any idea as to why it seems to have authenticated but it didnt? 

/yoandy