[jira] [Commented] (YARN-466) Slave hostname mismatches in ResourceManager/Scheduler

["Zhijie Shen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13742767#comment-13742767 ] 

Zhijie Shen commented on YARN-466:
----------------------------------

@Roger Hoover, I assume that you were talking about ApplicationMaster of MR, right?

1. Scheduler knows the hostname from RMNode, which is ultimately obtained from ContainerManagerImpl, which parses the hostname of the IPC server.

2. When MR job wants to allocate an container, it send the request, which can contain the hostname information from TaskAttemptImpl#dataLocalHosts. The hostname eventually comes from InputSplit. If it is FileInputFormat, it comes from the input files' getBlockLocations().

Still need to investigate more about how FileSystem name the hosts. If it uses the different method as NM does. It can happen that MRAppMaster sends the request with "aaaa", and Scheduler recognize the same host as "AAAA", leading to mismatch.
                
> Slave hostname mismatches in ResourceManager/Scheduler
> ------------------------------------------------------
>
>                 Key: YARN-466
>                 URL: https://issues.apache.org/jira/browse/YARN-466
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager, scheduler
>            Reporter: Roger Hoover
>            Assignee: Zhijie Shen
>
> The problem is that the ResourceManager learns the hostname of a slave node when the NodeManager registers itself and it seems the node manager is getting the hostname by asking the OS.  When a job is submitted, I think the ApplicationMaster learns the hostname by doing a reverse DNS lookup based on the slaves file.
> Therefore, the ApplicationMaster submits requests for containers using the fully qualified domain name (node1.foo.com) but the scheduler uses the OS hostname (node1) when checking to see if any requests are node-local.  The result is that node-local requests are never found using this method of searching for node-local requests:
> ResourceRequest request = application.getResourceRequest(priority, node.getHostName());
> I think it's unfriendly to ask users to make sure they configure hostnames to match fully qualified domain names. There should be a way for the ApplicationMaster and NodeManager to agree on the hostname.
> Steps to Reproduce:
> 1) Configure the OS hostname on slaves to differ from the fully qualified domain name.  For example, if the FQDN for the slave is "node1.foo.com", set the hostname on the node to be just "node1".
> 2) On submitting a job, observe that the AM submits resource requests using the FQDN (e.g. "node1.foo.com").  You can add logging to the allocate() method of whatever scheduler you're using 
> for (ResourceRequest req: ask) {
>       LOG.debug(String.format("Request %s for %d containers on %s", req, req.getNumContainers(), req.getHostName()));
>     }
> 3) Observe that when the scheduler checks for node locality (in the handle() method) using the FiCaSchedulerNode.getHostName(), the hostname is uses is the one set in the host OS (e.g. "node1").  NOTE: if you're using FifoScheduler, this bug needs to be fixed first (https://issues.apache.org/jira/browse/YARN-412).  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira