[jira] [Commented] (MAPREDUCE-2420) JobTracker should be able to renew delegation token over HTTP

["Devaraj Das (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016060#comment-13016060 ] 

Devaraj Das commented on MAPREDUCE-2420:
----------------------------------------

This patch handles the case of HDFS token renewals, when the JobTracker's HDFS is running a different version of HDFS than the one the job is trying to use. An example of such a job is distcp (where it uses hftp to talk to a different source cluster to pull data to the cluster where distcp is running).

When the job is submitted, the client requests a delegation token over hftp and stuffs it in the job. Today, the NameNode doesn't distinguish between hftp and hdfs accesses, and issues HDFS tokens for both (and the token-kind field in the token has the value as 'HDFS'). Ideally, that should be fixed to have the token-kind as HFTP for hftp accesses. We should have the JobTracker handle all sorts of token renewals, and have a way in which it can look at a token and decide which protocol to use to talk to the server in question. This includes HDFS, HFTP, and also HIVE (where the protocol is thrift). 

I think this patch is okay for the short term - the JobTracker falls back to hftp if it couldn't renew a token over hdfs. In the patch, there are a bunch of white space changes that aren't required. The string comparisons for exception messages and then instantiating a concrete exception could probably be replaced with a forname() on the string exception.

When we fix this issue in trunk, please make it more generic on lines similar to above.

> JobTracker should be able to renew delegation token over HTTP
> -------------------------------------------------------------
>
>                 Key: MAPREDUCE-2420
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2420
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.20.4
>            Reporter: Boris Shkolnik
>            Assignee: Boris Shkolnik
>         Attachments: MR-2420.patch
>
>
> in case JobTracker has to talk to a NameNode running a different version (RPC version mismatch), Jobtracker should be able to fall back to HTTP renewal.
> Example of the case - running distcp between different versions using hfpt.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira