[jira] Commented: (HADOOP-1448) Setting the replication factor of a file too high causes namenode cpu overload

["Hairong Kuang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12509953 ] 

Hairong Kuang commented on HADOOP-1448:
---------------------------------------

For the getBlockLocation operation, how about the following improvements?
1. For each block, the maximun number of replica locations returned is 3.
2. Instead of sorting, using the pseudo-sort in HADOOP-1155.  The returned list is in the order: local replica, local-rack replicas, and off-rack replicas. This algorithm does not require full linear scan of all replicas.
3. For the randomness, if there is a local copy, the first location is the local copy; otherwise if there are any local-rack copies, set the first location to be a random local-rack replica; otherwise, the first location is a random off-rack replica.

> Setting the replication factor of a file too high causes namenode cpu overload
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-1448
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1448
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>
> The replication factor of a file in set to 300 (on a 800 node cluster). Then all mappers try to open this file. For every open call that the namenode receives from each of these 800 clients, it sorts all the replicas of the block(s) based on the distance from the client. This causes CPU usage overload on the namenode.
> One proposal is to make the namenode return a non-sorted list of datanodes to the client. Information about each replica also contains the rack on which that replica resides. The client can look at the replicas to determine if there is a copy on the local node. If not, then it can find out if there is a replica on the local rack. If not then it can choose a replica at random.
> This proposal is scalable because the sorting and selection of replicas is done by the client rather than the Namenode.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.