You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by "Lorenzo Thione (JIRA)" <ji...@apache.org> on 2006/08/31 04:33:22 UTC

[jira] Created: (HADOOP-497) DataNodes and TaskTrackers should be able to report hostnames and ips relative to customizable network interfaces and nameservers

DataNodes and TaskTrackers should be able to report hostnames and ips relative to customizable network interfaces and nameservers
---------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-497
                 URL: http://issues.apache.org/jira/browse/HADOOP-497
             Project: Hadoop
          Issue Type: New Feature
          Components: dfs, mapred, util
    Affects Versions: 0.6.0
            Reporter: Lorenzo Thione
            Priority: Minor
         Attachments: dnsjava-2.0.2.jar, nif-utils.patch

This patch allows for network configuration parameters to be aded to the hadoop-site.xml file. These parameters specify a network interface name and an optional nameserver hostname which DataNodes and TaskTrackers consult to resolve  their hostnames from the IP bound to the specified network interface.

This is useful when machines that are part of different physical or logical network need to participate in hadoop clusters as client nodes. The hostname and IP reported by InetAddress.getLocalHost() are not necessarily the ones that will allow the JobTracker and NameNode to reach the clients, as well as not necessarily the ones through which the DFS clients can reach the DataNodes.

The configuration parameters are
 - cluster.report.nif
 - cluster.report.ns

nif: takes the name of a network interface, like en0, en1 (on macs), eth0, etc...
ns: the host name of a DNS server to use when resolving the IP bound to the specified nif

These parameters are set by default to the value "default" which will replicate the current behavior of reporting InetAddress.getLocalHost().getHostName() and getHostAddress()

As part of the patch, a new library dnsjava was added along with its license information (BSD license). The list of affected files is:

src
 org.apache.hadoop.dfs.DataNode 
 org.apache.hadoop.mapred.taskTracker 
 org.apache.hadoop.util.NetworkUtils
conf
 hadoop-default.xml
lib
 dnsjava-2.0.2.jar
 dnsjava-2.0.2.LICENSE.txt


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (HADOOP-497) DataNodes and TaskTrackers should be able to report hostnames and ips relative to customizable network interfaces and nameservers

Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.

    [ http://issues.apache.org/jira/browse/HADOOP-497?page=comments#action_12431926 ] 
            
Doug Cutting commented on HADOOP-497:
-------------------------------------

I'm not sure why you need the dnsjava library.  At a glance, it seems you could instead set sun.net.spi.nameservice.nameservers and then use the java.net.NetworkInterface to find which hostnames correspond to a particular interface.  Why doesn't that work?

A few nits with the code:

- It should probably be named net/DNS.java, insead of util/NetworkUtils.java.  The util package is already a mess.

- The descriptions of the new config properties only mention tasktracker and jobtracker, but the options are also valid for datanode and namenode.  I would also name these properties something like net.dns.interface and net.interface.nameserver.

- The new class and all its public members should have informative javadoc comments.  if we add a new package, it should have a package.html with at least a one-line description of the package.


> DataNodes and TaskTrackers should be able to report hostnames and ips relative to customizable network interfaces and nameservers
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-497
>                 URL: http://issues.apache.org/jira/browse/HADOOP-497
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred, dfs, util
>    Affects Versions: 0.6.0
>            Reporter: Lorenzo Thione
>            Priority: Minor
>         Attachments: dnsjava-2.0.2.jar, nif-utils.patch
>
>
> This patch allows for network configuration parameters to be aded to the hadoop-site.xml file. These parameters specify a network interface name and an optional nameserver hostname which DataNodes and TaskTrackers consult to resolve  their hostnames from the IP bound to the specified network interface.
> This is useful when machines that are part of different physical or logical network need to participate in hadoop clusters as client nodes. The hostname and IP reported by InetAddress.getLocalHost() are not necessarily the ones that will allow the JobTracker and NameNode to reach the clients, as well as not necessarily the ones through which the DFS clients can reach the DataNodes.
> The configuration parameters are
>  - cluster.report.nif
>  - cluster.report.ns
> nif: takes the name of a network interface, like en0, en1 (on macs), eth0, etc...
> ns: the host name of a DNS server to use when resolving the IP bound to the specified nif
> These parameters are set by default to the value "default" which will replicate the current behavior of reporting InetAddress.getLocalHost().getHostName() and getHostAddress()
> As part of the patch, a new library dnsjava was added along with its license information (BSD license). The list of affected files is:
> src
>  org.apache.hadoop.dfs.DataNode 
>  org.apache.hadoop.mapred.taskTracker 
>  org.apache.hadoop.util.NetworkUtils
> conf
>  hadoop-default.xml
> lib
>  dnsjava-2.0.2.jar
>  dnsjava-2.0.2.LICENSE.txt

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira