You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2007/01/26 01:27:49 UTC
[jira] Created: (HADOOP-940) pendingReplications of FSNamesystem is
not informative
pendingReplications of FSNamesystem is not informative
------------------------------------------------------
Key: HADOOP-940
URL: https://issues.apache.org/jira/browse/HADOOP-940
Project: Hadoop
Issue Type: Improvement
Components: dfs
Affects Versions: 0.10.1
Reporter: Hairong Kuang
Fix For: 0.11.0
Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-940) pendingReplications of FSNamesystem
is not informative
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12469923 ]
Hairong Kuang commented on HADOOP-940:
--------------------------------------
If a block is in neededReplication, it means that it needs more copies and needs to run chooseTarget. If a block is in pendingReplication, it means that chooseTarget has already been run and a Replication command has been send to datanodes.
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-940) pendingReplications of FSNamesystem
is not informative
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12469877 ]
dhruba borthakur commented on HADOOP-940:
-----------------------------------------
if a block is in pendingReplication, then it means that this block needs more copies. It does not say anythung about how many more copies are to be made. When a heartbeat arirves at the namenode, the heartbeat processing code picks up a block from neededReplication, determines how many new replicas to be made and then tells the datanode to start replication.
As part of hadoop-923, I am changing thet fact that heartbeat processing is doing all the heavy lifting. A new thread inside FSNamesystem will periodically pick up blocks from neededReplication and compute targets of new replicas.
But I agree that pendingReplication queue shoudl be periodically scanned and old items be dealt with appropriately.
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
> Fix For: 0.11.0
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-940) pendingReplications of FSNamesystem is
not informative
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-940:
------------------------------------
Attachment: pendingReplication2.patch
Changed the timeout period from 10 minutes to 5 minutes. Also, the periodic scanning for timed out requests occur every 5 minutes.
I still kept only one timer per block (instead of each replica of each block). You are right in stating that in this case we might incur twice the timeout period. But I would like to trade-off on not making the code more complex to handle that rare case.
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
> Attachments: pendingReplication2.patch
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-940) pendingReplications of FSNamesystem is
not informative
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-940:
--------------------------------
Resolution: Fixed
Fix Version/s: 0.12.0
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Dhruba!
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
> Assigned To: dhruba borthakur
> Fix For: 0.12.0
>
> Attachments: pendingReplication2.patch
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-940) pendingReplications of FSNamesystem is
not informative
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-940:
------------------------------------
Attachment: pendingReplication2.patch
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
> Assigned To: dhruba borthakur
> Attachments: pendingReplication2.patch
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-940) pendingReplications of FSNamesystem
is not informative
Posted by "Marco Nicosia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12469874 ]
Marco Nicosia commented on HADOOP-940:
--------------------------------------
+1 This problem is causing real issues on running clusters, affecting customers.
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
> Fix For: 0.11.0
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-940) pendingReplications of FSNamesystem
is not informative
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12469926 ]
Hairong Kuang commented on HADOOP-940:
--------------------------------------
The purpose of having targets in PendingReplication is for avoid overreplication. Having a timer associate with each target is to avoid under replication when the replication command gets lost or the target data node shuts down before sending the replication notification back to namenode.
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-940) pendingReplications of FSNamesystem
is not informative
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur reassigned HADOOP-940:
---------------------------------------
Assignee: dhruba borthakur
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
> Assigned To: dhruba borthakur
> Attachments: pendingReplication2.patch
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-940) pendingReplications of FSNamesystem
is not informative
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12476515 ]
Hadoop QA commented on HADOOP-940:
----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12352149/pendingReplication2.patch</a>) against trunk revision <a href= applied and successfully tested against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/512499. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
> Assigned To: dhruba borthakur
> Attachments: pendingReplication2.patch
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-940) pendingReplications of FSNamesystem is
not informative
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-940:
------------------------------------
Attachment: pendingReplication.patch
First version of code. Review comments welcome.
A new thread that records all replications that are currently in progress. If a replication request does not complete in 10 minutes, then the block is put back in neededReplication.
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
> Attachments: pendingReplication.patch
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-940) pendingReplications of FSNamesystem is
not informative
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-940:
------------------------------------
Attachment: (was: pendingReplication2.patch)
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
> Assigned To: dhruba borthakur
> Attachments: pendingReplication2.patch
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-940) pendingReplications of FSNamesystem
is not informative
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12476460 ]
Hadoop QA commented on HADOOP-940:
----------------------------------
-1, because javac generated 768 warnings (more than the acceptable 766 warnings) when testing the latest attachment (http://issues.apache.org/jira/secure/attachment/12352149/pendingReplication2.patch) against trunk revision http://svn.apache.org/repos/asf/lucene/hadoop/trunk/512461. Please note that this message is automatically generated and may represent a problem with the automation system and not the patch. Results are at http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
> Assigned To: dhruba borthakur
> Attachments: pendingReplication2.patch
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-940) pendingReplications of FSNamesystem is
not informative
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-940:
------------------------------------
Attachment: (was: pendingReplication.patch)
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
> Attachments: pendingReplication2.patch
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-940) pendingReplications of FSNamesystem is
not informative
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-940:
------------------------------------
Status: Patch Available (was: Open)
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
> Assigned To: dhruba borthakur
> Attachments: pendingReplication2.patch
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-940) pendingReplications of FSNamesystem
is not informative
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12476080 ]
Hairong Kuang commented on HADOOP-940:
--------------------------------------
This patch sets the time out period to be 10 minutes and the pending replication monitor thread scans the pending replication queue for time out blocks once every 1 minute. Because the chance that a block does not get replicated is rare, I feel that the monitor thread scans too frequently. Does it make more sense to set the time out period to be 5 minutes and the scanning frequency to be once every 5 minutes?
Each entry in the pendingReplication queue is a block with only one timer. When a replica is placed, the timer associated with the block is reset. This may cause that the re-replication of a replica gets delayed. For example, if a replica's replication request gets lost but another replica of the same block is placed right before its timer gets expired, the lost request won't be detected until twice of the timeout period is passed.
> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
> Key: HADOOP-940
> URL: https://issues.apache.org/jira/browse/HADOOP-940
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.10.1
> Reporter: Hairong Kuang
> Attachments: pendingReplication.patch
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to the pendingReplications queue. When it is no longer under replicated, it is pulled out of the pendingReplications queue. But the queue does not provide any information like how many targets have been choosen or who those targets are. PendingReplications are not used when deciding if a block is under replication. This may cause a block to be over replications or inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node is choosen to replicate this block and the block is put in the pendingReplications queue. If the block's replication factor is changed to be 3 before the block replication notification, which is the next block report, comes in, the block will be put into neededReplictions queue again under the assumption that it needs to choose 2 targets instead of 1. So the block will end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the total number of replicas that a block has. When the name node is notified that the block is replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to receive the block replication notification. PendingReplications queue needs to be periodically scanned to remove those data nodes whose timer is expired.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.