[jira] Commented: (HADOOP-692) Rack-aware Replica Placement

["Hairong Kuang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12463383 ] 

Hairong Kuang commented on HADOOP-692:
--------------------------------------

The attached patch includes:

1. data structures for describing cluster network topology
   a. org.apache.hadoop.net.NetworkTopology: class for manipulating the tree-like cluster topology.
   . InnerNode: represents racks/datacenters
   . Leaves: represents datanodes
   b. org.apache.hadoop.net.Node: class for abstracting a node in NetworkTopology
   . name: name of the node
   . networklocation: indicating which datacenter and rack that this node is located.
   c. org.apchae.hadoop.net.NodeBase: a base implementation of Node
2. code for implementing replica placement strategy
   a org.apache.hadoop.dfs.FSNamesystem.Replicator
   . chooseTarget(numOfReplicas, writer, excludedNodes, blockSize): choose numOfReplcas targets
   . chooseTarget(numOfReplicas, writer, choosenNodes, excludedNodes, block): repreplicate numOfReplicas targets
   Targets are returned forming a shortest path that starts from writer and traversing all targets
3. protocol changes
   a. org.apache.hadoop.dfs.ClientProtocol
   . open is enhanced by passing the clientMachine name. In the returned blocks, containing data nodes are sorted by its distance to the clientMachine
   b. org.apache.hadoop.dfs.DatanodeProtocol
   . register is enhanced by passing the data node network location. NetworkTopology is built while registering data nodes.
4. data structures changes to reflecting data node network location information
   a. org.apache.hadoop.dfs.Datanode
   b. org.apache.hadoop.dfs.DatanodeInfo
5. various test cases
   a. org.apache.hadoop.net.TestNetworkTopolgy: test NetworkTopology
   b. org.apache.hadoop.dfs.TestReplicationPolocy: test replicas placement
   c. org.apache.hadoop.dfs.TestRackAwareReplication: test replicas placement/datanode registration by bringing up a MiniDFSCluster
   d. org.apache.hadoop.dfs.MiniDFSCluster: enhance it to start datanodes on different network locations 

Please review! I appreciate your feedback.

> Rack-aware Replica Placement
> ----------------------------
>
>                 Key: HADOOP-692
>                 URL: https://issues.apache.org/jira/browse/HADOOP-692
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.8.0
>            Reporter: Hairong Kuang
>         Assigned To: Hairong Kuang
>         Attachments: rack.patch, Rack_aware_HDFS_proposal.pdf
>
>
> This issue assumes that HDFS runs on a cluster of computers that spread across many racks. Communication between two nodes on different racks needs to go through switches. Bandwidth in/out of a rack may be less than the total bandwidth of machines in the rack. The purpose of rack-aware replica placement is to improve data reliability, availability, and network bandwidth utilization. The basic idea is that each data node determines to which rack it belongs at the startup time and notifies the name node of the rack id upon registration. The name node maintains a rackid-to-datanode map and tries to place replicas across racks.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira