[jira] [Updated] (MAPREDUCE-4877) AM doesn't properly support multiple NNs

["Karthik Kambatla (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Karthik Kambatla updated MAPREDUCE-4877:
----------------------------------------

    Target Version/s: 2.6.0  (was: 3.0.0, 2.5.0)

> AM doesn't properly support multiple NNs
> ----------------------------------------
>
>                 Key: MAPREDUCE-4877
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4877
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster, job submission
>    Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>
> Yarn/MR clusters assume there's a 1-to-1 correspondence between itself and a NN.  Certain internal paths like the staging dir, job history, intermediate/intermediate-done dirs are resolved relative to the defaultFS.  The JT used the host's conf which ensured the correct/expected NN.  However the AM uses the user's job conf, which means the user's defined defaultFS can cause the job to use incorrect paths.
> Typically the output path's NN is also the yarn cluster's NN.  However problems occur when a yarn cluster is servicing multiple NN's (ex. federated clusters).  The JHS is assuming the AM will write to NN1, whereas the user's job conf may be using a defaultFS of NN2 or NN3 which influences where the AM writes.



--
This message was sent by Atlassian JIRA
(v6.2#6252)