Re: IP based hadoop cluster

[sodul <so...@odul.com>]

Replying to my own thread here.

While we got a good handle on the IP based hadoop cluster by using the
settings mentioned above, we are now upgrading the Cloudera 5.1.0 packages
and Yarn.

So far most everything seemed to work well, except that for some reason Yarn
insists on making use of DNS, which is causing issues in our environment.

We've set the following things in our yarn-site.xml (customized for each
host through automation):


The issue is that I still get java.net.NoRouteToHostException while running
jobs, and the exception is showing the fqdn of the hosts an not the IP
addresses which I want Yarn to use.

When looking at http://1.2.3.100:8042/conf I see that most values provided
in the xml are honored, but not all:


It seems to be a bug that yarn.nodemanager.localizer.address is set
'programatically' and not from yarn-site.xml. I have even added it as
-Dyarn.nodemanager.localizer.address=1.2.3.100:8040 to YARN_OPTS when
launching each individual node manager without any change shown in
http://1.2.3.100:8042/conf. 

My guess is that we need to be able to explicitaly what we want
yarn.nodemanager.localizer.address to be, but maybe also a way to tell the
ressourcemanager to not be 'smart' about the nodes fqdn, like
'hadoop.security.token.service.use_ip' is supposed to do.




--
View this message in context: http://hadoop.6.n7.nabble.com/IP-based-hadoop-cluster-tp70191p73503.html
Sent from the common-user mailing list archive at Nabble.com.