You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Nathan Roberts (JIRA)" <ji...@apache.org> on 2014/12/02 23:44:13 UTC
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer
preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232257#comment-14232257 ]
Nathan Roberts commented on HDFS-2825:
--------------------------------------
Any chance folks have altered their view on this and would accept a config option in the default placement policy? I'm happy to file the jira and put up the patch. It's very straightforward and makes sense for a lot of cases. imho, it actually makes more sense as the default than node-local (except for obvious things like Hbase). Today with node-local-first we wind up with some very hot nodes that dramatically slow down subsequent jobs, all because there is an entire copy of a large file on a single node of a rack.
> Add test hook to turn off the writer preferring its local DN
> ------------------------------------------------------------
>
> Key: HDFS-2825
> URL: https://issues.apache.org/jira/browse/HDFS-2825
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.24.0, 0.23.1
>
> Attachments: hdfs-2825.txt, hdfs-2825.txt
>
>
> Currently, the default block placement policy always places the first replica in the pipeline on the local node if there is a valid DN running there. In some network designs, within-rack bandwidth is never constrained so this doesn't give much of an advantage. It would also be really useful to disable this for MiniDFSCluster tests, since currently if you start a multi-DN cluster and write with replication level 1, all of the replicas go to the same DN.
> _[per discussion below, this was changed to not add a config, but only to add a hook for testing]_
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)