[jira] [Commented] (YARN-10857) YarnClient Caching Addresses

["Steve Suh (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-10857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384426#comment-17384426 ] 

Steve Suh commented on YARN-10857:
----------------------------------

An unresolved InetSocketAddress object is created during the initialization of the YarnClient.  This InetSocketAddress object is passed around and reused by the IPC [Client.java|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java]

Here the InetSocketAddress is only checked if it's an unresolved address.  It does not attempt to re-resolve the cached address and throws and error.
https://github.com/apache/hadoop/blob/de41ce8a16434aee13f705a9e3666f29e8ec8cb3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1602-L1608


{code:java}
    if (address.isUnresolved()) {
      throw NetUtils.wrapException(address.getHostName(),
          address.getPort(),
          null,
          0,
          new UnknownHostException());
    }
{code}



A possible fix would be to change this check and include the following: _if (address.isUnresolved() *{color:#4C9AFF}&& !updateAddress(){color}*)_
{code:java}
    if (address.isUnresolved() && !updateAddress()) {
      throw NetUtils.wrapException(address.getHostName(),
          address.getPort(),
          null,
          0,
          new UnknownHostException());
    }
{code}


> YarnClient Caching Addresses
> ----------------------------
>
>                 Key: YARN-10857
>                 URL: https://issues.apache.org/jira/browse/YARN-10857
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: client, yarn
>            Reporter: Steve Suh
>            Assignee: Prabhu Joseph
>            Priority: Minor
>
> We have noticed that when the YarnClient is initialized and used, it is not very resilient when dns or /etc/hosts is modified in the following scenario:
> Take for instance the following (and reproducable) sequence of events that can occur on a service that instantiates and uses YarnClient. 
>   - Yarn has rm HA enabled (*yarn.resourcemanager.ha.enabled* is *true*) and there are two rms (rm1 and rm2).
>   - *yarn.client.failover-proxy-provider* is set to *org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider*
> 1)	rm2 is currently the active rm
> 2)	/etc/hosts (or dns) is missing host information for rm2
> 3)	A service is started and it initializes the YarnClient at startup.
> 4)	At some point in time after YarnClient is done initializing, /etc/hosts is updated and contains host information for rm2
> 5)	Yarn is queried, for instance calling *yarnclient.getApplications()*
> 6)	All YarnClient attempts to communicate with rm2 fail with UnknownHostExceptions, even though /etc/hosts now contains host information for it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org