You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2014/08/13 19:58:12 UTC
[jira] [Commented] (HDFS-6846) NetworkTopology#sortByDistance
should give nodes higher priority, which cache the block.
[ https://issues.apache.org/jira/browse/HDFS-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095807#comment-14095807 ]
Jason Lowe commented on HDFS-6846:
----------------------------------
This could be very undesirable if a single node is the only one that has a cached block and suddenly the block becomes very popular (e.g.: during localization across many nodes in a large cluster). Unless the block is highly replicated, most requests will be off-rack and the one node that has it cached will be hammered. Having the block in memory doesn't help if the NIC saturates from the traffic. I just want to make sure we don't end up with another form of HDFS-6840.
> NetworkTopology#sortByDistance should give nodes higher priority, which cache the block.
> ----------------------------------------------------------------------------------------
>
> Key: HDFS-6846
> URL: https://issues.apache.org/jira/browse/HDFS-6846
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Affects Versions: 2.6.0
> Reporter: Yi Liu
> Assignee: Yi Liu
>
> Currently there are 3 weights:
> * local
> * same rack
> * off rack
> But if some nodes cache the block, then it's faster if client read block from these nodes. So we should have some more weights as following:
> * local
> * cached & same rack
> * same rack
> * cached & off rack
> * off rack
--
This message was sent by Atlassian JIRA
(v6.2#6252)