You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2014/08/13 19:58:12 UTC

[jira] [Commented] (HDFS-6846) NetworkTopology#sortByDistance should give nodes higher priority, which cache the block.

    [ https://issues.apache.org/jira/browse/HDFS-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095807#comment-14095807 ] 

Jason Lowe commented on HDFS-6846:
----------------------------------

This could be very undesirable if a single node is the only one that has a cached block and suddenly the block becomes very popular (e.g.: during localization across many nodes in a large cluster).  Unless the block is highly replicated, most requests will be off-rack and the one node that has it cached will be hammered.  Having the block in memory doesn't help if the NIC saturates from the traffic.  I just want to make sure we don't end up with another form of HDFS-6840.

> NetworkTopology#sortByDistance should give nodes higher priority, which cache the block.
> ----------------------------------------------------------------------------------------
>
>                 Key: HDFS-6846
>                 URL: https://issues.apache.org/jira/browse/HDFS-6846
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.6.0
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>
> Currently there are 3 weights:
> * local
> * same rack
> * off rack
> But if some nodes cache the block, then it's faster if client read block from these nodes. So we should have some more weights as following:
> * local
> * cached & same rack
> * same rack
> * cached & off rack
> * off rack



--
This message was sent by Atlassian JIRA
(v6.2#6252)