You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Russell Brown <ru...@oracle.com> on 2010/06/14 19:57:46 UTC
Hadoop and IP on InfiniBand (IPoIB)
I'm a new user of Hadoop. I have a Linux cluster with both gigabit
ethernet and InfiniBand communications interfaces. Could someone please
tell me how to switch IP communication from ethernet (the default) to
InfiniBand? Thanks.
--
------------------------------------------------------------
Russell A. Brown | Oracle
russ.brown@oracle.com | UMPK14-260
(650) 786-3011 (office) | 14 Network Circle
(650) 786-3453 (fax) | Menlo Park, CA 94025
------------------------------------------------------------
Re: Hadoop and IP on InfiniBand (IPoIB)
Posted by Russell Brown <ru...@oracle.com>.
FYI, Allen Wittnauer,
I'm using Linux not Solaris, but I'll pay attention to your comment
about Solaris if I install Solaris on the cluster. Thanks again for
your helpful comments.
Russ
On 06/15/10 11:10 AM, Allen Wittenauer wrote:
> On Jun 15, 2010, at 7:40 AM, Russell Brown wrote:
>
>
>> Thanks, Allen, for responding.
>>
>> So, if I understand you correctly, the dfs.datanode.dns.interface and mapred.tasktracker.dns.interface options may be used to define inbound connections only?
>>
>
> Correct. The daemons will bind to those interfaces and use those names as their 'official' connection in.
>
>
>> Concerning the OS configuration, my /etc/hosts files assign unique host names to the ethernet and IB interfaces. However, even if I specify the IB host names in the masters and slaves files, communication still occurs via ethernet, not via IB.
>>
>
> BTW, are you doing this on Solaris or Linux?
>
> Solaris is notorious for not honoring inbound and outbound interfaces. [In other words, just because the packet came in on bge0, that is no guarantee that the reply will go out on bge0 if another route is available. Particularly frustrating with NFS and SunCluster.]
>
>
>> Your recommendation would therefore be to define IB instead of ethernet as the default network interface connection, right?
>>
>
> Yup. Or at least give it a lower cost in the routing table.
--
------------------------------------------------------------
Russell A. Brown | Oracle
russ.brown@oracle.com | UMPK14-260
(650) 786-3011 (office) | 14 Network Circle
(650) 786-3453 (fax) | Menlo Park, CA 94025
------------------------------------------------------------
Re: Hadoop and IP on InfiniBand (IPoIB)
Posted by Allen Wittenauer <aw...@linkedin.com>.
On Jun 15, 2010, at 7:40 AM, Russell Brown wrote:
> Thanks, Allen, for responding.
>
> So, if I understand you correctly, the dfs.datanode.dns.interface and mapred.tasktracker.dns.interface options may be used to define inbound connections only?
Correct. The daemons will bind to those interfaces and use those names as their 'official' connection in.
> Concerning the OS configuration, my /etc/hosts files assign unique host names to the ethernet and IB interfaces. However, even if I specify the IB host names in the masters and slaves files, communication still occurs via ethernet, not via IB.
BTW, are you doing this on Solaris or Linux?
Solaris is notorious for not honoring inbound and outbound interfaces. [In other words, just because the packet came in on bge0, that is no guarantee that the reply will go out on bge0 if another route is available. Particularly frustrating with NFS and SunCluster.]
> Your recommendation would therefore be to define IB instead of ethernet as the default network interface connection, right?
Yup. Or at least give it a lower cost in the routing table.
Re: Hadoop and IP on InfiniBand (IPoIB)
Posted by Russell Brown <ru...@oracle.com>.
Thanks, Allen, for responding.
So, if I understand you correctly, the dfs.datanode.dns.interface and
mapred.tasktracker.dns.interface options may be used to define inbound
connections only?
Concerning the OS configuration, my /etc/hosts files assign unique host
names to the ethernet and IB interfaces. However, even if I specify the
IB host names in the masters and slaves files, communication still
occurs via ethernet, not via IB.
Your recommendation would therefore be to define IB instead of ethernet
as the default network interface connection, right?
Thanks,
Russ
On 06/14/10 12:32 PM, Allen Wittenauer wrote:
> On Jun 14, 2010, at 10:57 AM, Russell Brown wrote:
>
>
>> I'm a new user of Hadoop. I have a Linux cluster with both gigabit ethernet and InfiniBand communications interfaces. Could someone please tell me how to switch IP communication from ethernet (the default) to InfiniBand? Thanks.
>>
>
>
> Hadoop will bind inbound connections via the interface settings in the various hadoop configuration files. Outbound connections are unbound and based solely on OS configuration. I filed a jira to fix this, but it is obviously low priority since few people run multi-nic boxes. Best bet is to down the ethernet and up the IB, changing routing, etc, as necessary.
--
------------------------------------------------------------
Russell A. Brown | Oracle
russ.brown@oracle.com | UMPK14-260
(650) 786-3011 (office) | 14 Network Circle
(650) 786-3453 (fax) | Menlo Park, CA 94025
------------------------------------------------------------
Re: Hadoop and IP on InfiniBand (IPoIB)
Posted by Allen Wittenauer <aw...@linkedin.com>.
On Jun 14, 2010, at 10:57 AM, Russell Brown wrote:
> I'm a new user of Hadoop. I have a Linux cluster with both gigabit ethernet and InfiniBand communications interfaces. Could someone please tell me how to switch IP communication from ethernet (the default) to InfiniBand? Thanks.
Hadoop will bind inbound connections via the interface settings in the various hadoop configuration files. Outbound connections are unbound and based solely on OS configuration. I filed a jira to fix this, but it is obviously low priority since few people run multi-nic boxes. Best bet is to down the ethernet and up the IB, changing routing, etc, as necessary.