You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Allen Wittenauer (JIRA)" <ji...@apache.org> on 2015/03/10 02:49:38 UTC
[jira] [Updated] (HDFS-1962) Enhance MiniDFSCluster to improve
testing of network topology distance related issues.
[ https://issues.apache.org/jira/browse/HDFS-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated HDFS-1962:
-----------------------------------
Fix Version/s: (was: 0.24.0)
> Enhance MiniDFSCluster to improve testing of network topology distance related issues.
> --------------------------------------------------------------------------------------
>
> Key: HDFS-1962
> URL: https://issues.apache.org/jira/browse/HDFS-1962
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.22.0
> Reporter: Eric Payne
>
> In Jira HDFS-1875, Tanping Wang added the following comment. In order to keep the scope of HDFS-1875 small, I have created this Jira to capture this need.
> -------------------------------------------------
> It would be really useful if we can have multiple simulated data nodes binded to different hosts and dfs client binded to a particular host. And futher down the road, some of the simulated data nodes on different hosts, but the same rack. We can use this to test network topology distance related issues.
> One of the related problem that I ran into was that the order of data nodes in LocatedBlock returned by name nodes is sorted by NetworkTopology#pseudoSortByDistance(). In current Mini dfs cluster, there is no way I can bind the client to a host or bind a simulated data node to a particular host/rack. It would be nice if mini dfs cluster can make this possible, so that the network topology distance of client to each data node is fixed. Therefore, the order of data nodes returned within a LocatedBlock on MiniDFS cluster is fixed. Currently the order of data nodes in LocatedBlock is randomly sorted which means NetworkTopology understand the DFSClient and simulated datanodes are not different hosts and different racks.
> Also in currently Mini DFS client provides the option of -racks when starting data nodes. But we can not bind multiple simulated data nodes to one rack... so it is not really that useful.
> -------------------------------------------------
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)