[jira] [Commented] (YARN-1994) Expose YARN/MR endpoints on multiple interfaces

["Craig Welch (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055314#comment-14055314 ] 

Craig Welch commented on YARN-1994:
-----------------------------------

We plan for our approach to be similar to the hdfs approach - there will be a service level bind-host parameter per service, like this:

<property>
    <name>yarn.resourcemanager.bind-host</name>
    <value>0.0.0.0</value>
  </property>

  <property>
    <name>yarn.nodemanager.bind-host</name>
    <value>0.0.0.0</value>
  </property>

this will control the address component of the binding for the listening processes for these services.  Any address (ip or name) which the host running the service can map to one of it's interfaces would be valid, but it's expected that it will primarily be used to indicate the desire to listen on all interfaces, which the 0.0.0.0 value above will accomplish.  Where .address values for service endpoints are configured in the yarn-site.conf the port values will be taken from there - if not specified the default ports will be used (this is to allow one bind specification for the multiple port-listeners of these services, rpc and webapp, etc).  

For muti-homed host support this would need to be combined with dfs (hdfs) mutli-home configuration (see hdfs patches referenced above, there is an additional configuration required for hdfs, see below) and the hosts will need to resolve to the same name on each of the networks they are attached to.  So, if a host was "hadoop1" on an internal network with address 10.0.0.22, which is the primary network where the cluster transfers it's data, etc, and there was a desire to support an external 10.0.1.* network for client access to submit jobs, talk to hdfs, view the management webapp pages, logs, etc, it would also need to resolve as "hadoop1" on that "management" network (say, 10.0.1.22 (though there is no reason the last value of the ip must be the same, it's only the case that the same name has to resolve properly on both networks, and where ever addresses for a host are specified in yarn-site.conf, etc, they must use that name, not a different name or an ip). 

An hdfs-site.xml configuration is required for multi-home hdfs support in addition to the bind properties in the hdfs patches referenced above, it is:

<property>
<name>dfs.client.use.datanode.hostname</name>
<value>true</value>
</property>


> Expose YARN/MR endpoints on multiple interfaces
> -----------------------------------------------
>
>                 Key: YARN-1994
>                 URL: https://issues.apache.org/jira/browse/YARN-1994
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager, resourcemanager, webapp
>    Affects Versions: 2.4.0
>            Reporter: Arpit Agarwal
>            Assignee: Craig Welch
>
> YARN and MapReduce daemons currently do not support specifying a wildcard address for the server endpoints. This prevents the endpoints from being accessible from all interfaces on a multihomed machine.
> Note that if we do specify INADDR_ANY for any of the options, it will break clients as they will attempt to connect to 0.0.0.0. We need a solution that allows specifying a hostname or IP-address for clients while requesting wildcard bind for the servers.
> (List of endpoints is in a comment below)



--
This message was sent by Atlassian JIRA
(v6.2#6252)