You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Steve Suh (Jira)" <ji...@apache.org> on 2021/07/16 20:15:00 UTC

[jira] [Updated] (YARN-10857) YarnClient Caching Addresses

     [ https://issues.apache.org/jira/browse/YARN-10857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Suh updated YARN-10857:
-----------------------------
    Description: 
We have noticed that when the YarnClient is initialized and used, it is not very resilient when dns or /etc/hosts is modified in the following scenario:

Take for instance the following (and reproducable) sequence of events that can occur on a service that instantiates and uses YarnClient. 
  - Yarn has rm HA enabled (*yarn.resourcemanager.ha.enabled* is *true*) and there are two rms (rm1 and rm2).
  - *yarn.client.failover-proxy-provider* is set to *org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider*

1)	rm2 is currently the active rm
2)	/etc/hosts (or dns) is missing host information for rm2
3)	A service is started and it initializes the YarnClient at startup.
4)	At some point in time after YarnClient is done initializing, /etc/hosts is updated and contains host information for rm2
5)	Yarn is queried using YarnClient, for instance calling `.getApplications()`
6)	All YarnClient attempts to communicate with rm2 fail with UnknownHostExceptions, even though /etc/hosts now contains host information for it.



  was:
We have noticed that when the YarnClient is initialized and used, it is not very resilient when dns or /etc/hosts is modified in the following scenario:

Take for instance the following (and reproducable) sequence of events that can occur on a service that instantiates and uses YarnClient. 
  - Yarn has rm HA enabled (`yarn.resourcemanager.ha.enabled` is `true`) and there are two rms (rm1 and rm2).
  - `yarn.client.failover-proxy-provider` is set to `org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider`

1)	rm2 is currently the active rm
2)	/etc/hosts (or dns) is missing host information for rm2
3)	A service is started and it initializes the YarnClient at startup.
4)	At some point in time after YarnClient is done initializing, /etc/hosts is updated and contains host information for rm2
5)	Yarn is queried using YarnClient, for instance calling `.getApplications()`
6)	All YarnClient attempts to communicate with rm2 fail with UnknownHostExceptions, even though /etc/hosts now contains host information for it.




> YarnClient Caching Addresses
> ----------------------------
>
>                 Key: YARN-10857
>                 URL: https://issues.apache.org/jira/browse/YARN-10857
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: client, yarn
>            Reporter: Steve Suh
>            Priority: Minor
>
> We have noticed that when the YarnClient is initialized and used, it is not very resilient when dns or /etc/hosts is modified in the following scenario:
> Take for instance the following (and reproducable) sequence of events that can occur on a service that instantiates and uses YarnClient. 
>   - Yarn has rm HA enabled (*yarn.resourcemanager.ha.enabled* is *true*) and there are two rms (rm1 and rm2).
>   - *yarn.client.failover-proxy-provider* is set to *org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider*
> 1)	rm2 is currently the active rm
> 2)	/etc/hosts (or dns) is missing host information for rm2
> 3)	A service is started and it initializes the YarnClient at startup.
> 4)	At some point in time after YarnClient is done initializing, /etc/hosts is updated and contains host information for rm2
> 5)	Yarn is queried using YarnClient, for instance calling `.getApplications()`
> 6)	All YarnClient attempts to communicate with rm2 fail with UnknownHostExceptions, even though /etc/hosts now contains host information for it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org