You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Ning Zhang (JIRA)" <ji...@apache.org> on 2009/11/21 02:39:39 UTC
[jira] Commented: (HDFS-782) dynamic replication
[ https://issues.apache.org/jira/browse/HDFS-782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780872#action_12780872 ]
Ning Zhang commented on HDFS-782:
---------------------------------
To elaborate on the proposal, a data node keeps the statistics on how many clients are requesting a certain block. If the number exceeds a certain threshold, the data node can send the block to a number of data nodes (children) and ask them to replicate the block (one heuristics is to choose from the data nodes whose asked for the block). If a child data node accepts the replication request (e.g., it doesn't hold already), it goes through the same protocol as adding a new replica acknowledged by the name node. The reason we propose datanode->datanode replication rather than datanode->namenode->datanode replication is that it is much faster for the former case than the latter (whose performance depending on the work load of the name node could be minutes). If the children also got too many requests, they can proactively replicate themselves recursively, until the # of requests are distributed to sufficient number of replicas.
Currently the name node cleans up the extra replicas periodically. To address DN->DN dynamic replication, we need to add a heuristic to let it clean extra replicas only when they has not been access in a certain period.
Any suggests?
> dynamic replication
> -------------------
>
> Key: HDFS-782
> URL: https://issues.apache.org/jira/browse/HDFS-782
> Project: Hadoop HDFS
> Issue Type: New Feature
> Reporter: Ning Zhang
>
> In a large and busy cluster, a block can be requested by many clients at the same time. HDFS-767 tries to solve the failing case when the # of retries exceeds the maximum # of retries. However, that patch doesn't solve the performance issue since all failing clients have to wait a certain period before retry, and the # of retries could be high.
> One solution to solve the performance issue is to increase the # of replicas for this "hot" block dynamically when it is requested many times at a short period. The name node need to be aware such situation and only clean up extra replicas when they are not accessed recently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.