[jira] [Created] (MAPREDUCE-4877) AM doesn't properly support multiple NNs

["Daryn Sharp (JIRA)" <ji...@apache.org>]

Daryn Sharp created MAPREDUCE-4877:
--------------------------------------

             Summary: AM doesn't properly support multiple NNs
                 Key: MAPREDUCE-4877
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4877
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: applicationmaster, job submission
    Affects Versions: 2.0.0-alpha, 0.23.0, 3.0.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


Yarn/MR clusters assume there's a 1-to-1 correspondence between itself and a NN.  Certain internal paths like the staging dir, job history, intermediate/intermediate-done dirs are resolved relative to the defaultFS.  The JT used the host's conf which ensured the correct/expected NN.  However the AM uses the user's job conf, which means the user's defined defaultFS can cause the job to use incorrect paths.

Typically the output path's NN is also the yarn cluster's NN.  However problems occur when a yarn cluster is servicing multiple NN's (ex. federated clusters).  The JHS is assuming the AM will write to NN1, whereas the user's job conf may be using a defaultFS of NN2 or NN3 which influences where the AM writes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira