You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Johannes Zillmann <jz...@googlemail.com> on 2010/05/28 12:24:12 UTC
DNS.reverseDns() fallback
Hi guys,
i have following situation. A network setup with 2 network interfaces, eth0 (external) and eth1(internal).
Now in order to use the internal ips for hadoop i set dfs.datanode.dns.interface and mapred.tasktracker.dns.interface to eth1.
When tasktracker or datanode starting the following happens:
- hadoop picks the ip(s) from eth1
- hadoop tries to do a reverse dns lookup on the internal ip(s) - which fails
- hadoop fallback to InetAddress.getLocalHost() - which is the external dns
So in the end DNS.getDefaultIp(eth1) returns the internal ip whereas DNS.getDefaultHost(eth1) returns the external dns.
Isn't this a bug. Should in case the reverse lookup fails InetAddress.getByName(ips[ctr]).getCanonicalHost() rather be taken ?
Are there any suggested workarounds for this situation ?
best regards
Johannes
Re: DNS.reverseDns() fallback
Posted by Johannes Zillmann <jz...@googlemail.com>.
Thanks for your input!
I added an issue for improving the fallback mechanism: https://issues.apache.org/jira/browse/HADOOP-6797
Johannes
On May 28, 2010, at 7:46 PM, Michael Segel wrote:
>
>
>
>> Date: Fri, 28 May 2010 13:13:56 -0400
>> Subject: Re: DNS.reverseDns() fallback
>> From: esammer@cloudera.com
>> To: common-user@hadoop.apache.org
>> CC: core-user@hadoop.apache.org
> [SNIP]
>>> IMHO, after looking at this issue, it really doesn't matter since the cloud shouldn't be getting a lot of external traffic except on the name node/job tracker nodes which could be multi-homed.
>>
>> It might be useful in the case where you're streaming data off of HDFS
>> directly to clients rather than in the MR or HBase case. Data import /
>> export comes to mind. Remember that clients establish a direct
>> connection to data nodes so a multihomed NN is insufficient. In that
>> case, "external" doesn't necessarily mean a public (routable) IP, but
>> simply another network. We've seen use cases for this in some
>> installations. One example is a data aggregation or ingestion network
>> is separate from the Hadoop internal network and you'd like to get
>> data into HDFS.
>>
>
> Eric,
> I don't disagree that it is useful.
>
> I was just offering up an alternative cloud design which would work as a work-around.
> As you point out, clients need to talk to all of the nodes.
> So you can make your name node your gateway for the rest of the cloud.
> (not a great idea but it works.)
> Or you can launch your jobs from a client or clients, who are multi-homed.
>
> If you're a Cloudera fan, you could use the cloudera desktop to launch jobs.
> So as long as that machine is visible to the cloud, you'll be OK too. ;-)
>
> I don't really recommend this, but its still a viable solution.
>
> -Mike
>
>
>
> _________________________________________________________________
> Hotmail is redefining busy with tools for the New Busy. Get more from your inbox.
> http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_2
RE: DNS.reverseDns() fallback
Posted by Michael Segel <mi...@hotmail.com>.
> Date: Fri, 28 May 2010 13:13:56 -0400
> Subject: Re: DNS.reverseDns() fallback
> From: esammer@cloudera.com
> To: common-user@hadoop.apache.org
> CC: core-user@hadoop.apache.org
[SNIP]
> > IMHO, after looking at this issue, it really doesn't matter since the cloud shouldn't be getting a lot of external traffic except on the name node/job tracker nodes which could be multi-homed.
>
> It might be useful in the case where you're streaming data off of HDFS
> directly to clients rather than in the MR or HBase case. Data import /
> export comes to mind. Remember that clients establish a direct
> connection to data nodes so a multihomed NN is insufficient. In that
> case, "external" doesn't necessarily mean a public (routable) IP, but
> simply another network. We've seen use cases for this in some
> installations. One example is a data aggregation or ingestion network
> is separate from the Hadoop internal network and you'd like to get
> data into HDFS.
>
Eric,
I don't disagree that it is useful.
I was just offering up an alternative cloud design which would work as a work-around.
As you point out, clients need to talk to all of the nodes.
So you can make your name node your gateway for the rest of the cloud.
(not a great idea but it works.)
Or you can launch your jobs from a client or clients, who are multi-homed.
If you're a Cloudera fan, you could use the cloudera desktop to launch jobs.
So as long as that machine is visible to the cloud, you'll be OK too. ;-)
I don't really recommend this, but its still a viable solution.
-Mike
_________________________________________________________________
Hotmail is redefining busy with tools for the New Busy. Get more from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_2
Re: DNS.reverseDns() fallback
Posted by Eric Sammer <es...@cloudera.com>.
On Fri, May 28, 2010 at 12:06 PM, Michael Segel
<mi...@hotmail.com> wrote:
>
> Hi,
>
> You can't do that.
Unfortunately, Mike is right.
> The problem is that Hadoop is going to pick up your external IP address because that's what the machine name resolves to. Then your slave nodes are on the internal route and you don't see them.
>
> Is it a bug? Maybe. More like a design defect.
Definitely in the design defect category. The host name handling /
binding code is... complicated and not ideal for these types of
situations.
> The work around is to forget about using the second nic card for hadoop/hbase traffic. Or make the internal network match your machine name and its dns information. Then use the second ip address to communicate with the outside world.
>
> So if your machine name is foo, and your dominiant ip address is on eth(0), you want foo.company.com to resolve to eth(0) and foo-ext.company.com
> to resolve to eth(1). Its backwards but it should work.
>
> IMHO, after looking at this issue, it really doesn't matter since the cloud shouldn't be getting a lot of external traffic except on the name node/job tracker nodes which could be multi-homed.
It might be useful in the case where you're streaming data off of HDFS
directly to clients rather than in the MR or HBase case. Data import /
export comes to mind. Remember that clients establish a direct
connection to data nodes so a multihomed NN is insufficient. In that
case, "external" doesn't necessarily mean a public (routable) IP, but
simply another network. We've seen use cases for this in some
installations. One example is a data aggregation or ingestion network
is separate from the Hadoop internal network and you'd like to get
data into HDFS.
--
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com
RE: DNS.reverseDns() fallback
Posted by Michael Segel <mi...@hotmail.com>.
Hi,
You can't do that.
The problem is that Hadoop is going to pick up your external IP address because that's what the machine name resolves to. Then your slave nodes are on the internal route and you don't see them.
Is it a bug? Maybe. More like a design defect.
The work around is to forget about using the second nic card for hadoop/hbase traffic. Or make the internal network match your machine name and its dns information. Then use the second ip address to communicate with the outside world.
So if your machine name is foo, and your dominiant ip address is on eth(0), you want foo.company.com to resolve to eth(0) and foo-ext.company.com
to resolve to eth(1). Its backwards but it should work.
IMHO, after looking at this issue, it really doesn't matter since the cloud shouldn't be getting a lot of external traffic except on the name node/job tracker nodes which could be multi-homed.
HTH
-Mike
PS. If I'm wrong please let me know. I had this problem and at the time, no solution was available.
> Subject: DNS.reverseDns() fallback
> From: jzillmann@googlemail.com
> Date: Fri, 28 May 2010 12:24:12 +0200
> To: core-user@hadoop.apache.org
>
> Hi guys,
>
> i have following situation. A network setup with 2 network interfaces, eth0 (external) and eth1(internal).
> Now in order to use the internal ips for hadoop i set dfs.datanode.dns.interface and mapred.tasktracker.dns.interface to eth1.
> When tasktracker or datanode starting the following happens:
> - hadoop picks the ip(s) from eth1
> - hadoop tries to do a reverse dns lookup on the internal ip(s) - which fails
> - hadoop fallback to InetAddress.getLocalHost() - which is the external dns
>
> So in the end DNS.getDefaultIp(eth1) returns the internal ip whereas DNS.getDefaultHost(eth1) returns the external dns.
>
> Isn't this a bug. Should in case the reverse lookup fails InetAddress.getByName(ips[ctr]).getCanonicalHost() rather be taken ?
> Are there any suggested workarounds for this situation ?
>
> best regards
> Johannes
>
_________________________________________________________________
Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_1