[jira] [Updated] (HDFS-6536) FileSystem.Cache.closeAll() throws authentication exception at the end of a webhdfs client

["Yongjun Zhang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yongjun Zhang updated HDFS-6536:
--------------------------------

    Summary: FileSystem.Cache.closeAll() throws authentication exception at the end of a webhdfs client  (was: FileSystem.Cache.closeAll() threw an exception due to authentication failure at the end of a webhdfs client session)

> FileSystem.Cache.closeAll() throws authentication exception at the end of a webhdfs client
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-6536
>                 URL: https://issues.apache.org/jira/browse/HDFS-6536
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client, webhdfs
>    Affects Versions: 2.4.0
>            Reporter: Yongjun Zhang
>
> With a small client program below, when running as user "root", exception is thrown at the end of the client run. The config is HA security enabled, with client config setting 
> {code}
>   <property>
>     <name>fs.defaultFS</name>
>     <value>webhdfs://ns1</value>
>   </property>
> {code}
> The client program:
> {code}
> public class kclient1 {
> 	public static void main(String[] args) throws IOException {
> 	  final Configuration conf = new Configuration();
>           //a non-root user
>           final UserGroupInformation ugi = UserGroupInformation.getUGIFromTicketCache("/tmp/krb5cc_496", "hdfs@xyz.com");  
>           System.out.println("Starting");
>           ugi.doAs(new PrivilegedAction<Object>() {
> 	  @Override
> 	      public Object run() {
> 		  try {
> 		  	FileSystem fs = FileSystem.get(conf);			
> 		  	String renewer = "abcdefg";
> 		  	fs.addDelegationTokens(
> 		  			renewer, ugi.getCredentials());
> 		  	// Just to prove that we connected with right credentials.
> 		  	fs.getFileStatus(new Path("/"));
> 		  	return fs.getDelegationToken(renewer);
> 		  } catch (Exception e) {
> 		  	e.printStackTrace();
> 		  	return null;
>                   }
>            }
>     });
>     System.out.println("THE END");
>   }
> }		
> {code}
> Output:
> {code}
> [root@yjzc5w-1 tmp2]# hadoop --config /tmp2/conf jar kclient1.jar kclient1.kclient1
> Starting
> 14/06/14 20:38:51 WARN ssl.FileBasedKeyStoresFactory: The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded
> 14/06/14 20:38:52 INFO web.WebHdfsFileSystem: Retrying connect to namenode: yjzc5w-2.xyz.com/172.26.3.87:20101. Already tried 0 time(s); retry policy is org.apache.hadoop.io.retry.RetryPolicies$FailoverOnNetworkExceptionRetry@1a92210, delay 0ms.
> To prove that connection with right credentials
> to get file status updated updated 7
> THE END
> 14/06/14 20:38:53 WARN ssl.FileBasedKeyStoresFactory: The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded
> 14/06/14 20:38:53 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
> 14/06/14 20:38:53 INFO fs.FileSystem: FileSystem.Cache.closeAll() threw an exception:
> java.io.IOException: Authentication failed, url=http://yjzc5w-2.xyz.com:20101/webhdfs/v1/?op=CANCELDELEGATIONTOKEN&user.name=root&token=HAAEaGRmcwRoZGZzAIoBRp2bNByKAUbBp7gcbBQUD6vWmRYJRv03XZj7Jajf8PU8CB8SV0VCSERGUyBkZWxlZ2F0aW9uC2hhLWhkZnM6bnMx
> [root@yjzc5w-1 tmp2]# 
> {code}
> We can see the the exception is thrown in the end of the client run.
> I found that the problem is that at the end of client run, like the C++ destructor is called at the end of object scope, the tokens stored in the filesystem cache is get cancelled with the following all:
> {code}
> final class TokenAspect<T extends FileSystem & Renewable> {
>   @InterfaceAudience.Private
>   public static class TokenManager extends TokenRenewer {
>     @Override
>     public void cancel(Token<?> token, Configuration conf) throws IOException {
>       getInstance(token, conf).cancelDelegationToken(token); <==
>     }
> {code}
> where getInstance(token, conf) create a FileSystem as user "root", then call cancelDelegationToken to server side. However, server doesn't have "root" credential, so throw this exceptoin.
> When I run the same program as user "hdfs", then it's fine.
> I think if we run the call to cancelDelegationToken as the user who created the token intially ("hdfs" in this case), then it should work fine.
> However, the information of the user who created the token is not available at that point. 
> Hi [~daryn], I wonder if you could give a quick comment, really appreciate it!



--
This message was sent by Atlassian JIRA
(v6.2#6252)