You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Junping Du (JIRA)" <ji...@apache.org> on 2012/09/16 21:07:08 UTC
[jira] [Created] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Junping Du created HDFS-3941:
--------------------------------
Summary: Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
Key: HDFS-3941
URL: https://issues.apache.org/jira/browse/HDFS-3941
Project: Hadoop HDFS
Issue Type: New Feature
Components: name-node
Affects Versions: 1.0.0
Reporter: Junping Du
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao updated HDFS-3941:
----------------------------
Attachment: HDFS-3941.003.patch
Junping, thanks for the comments! You're right, my 002 patch updates numOfAvailableNodes. The new 003 patch removes this part, also remove part of the testcases from TestReplicationPolicyWithNodeGroup which are not included in the trunk currently.
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.002.patch, HDFS-3941.003.patch, HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao updated HDFS-3941:
----------------------------
Attachment: HDFS-3941.002.patch
Based on Junping's original patch, I tried to address Nicholas's comments and generated a new patch which may be more close to the current trunk.
Junping, if you have not worked on this, could you please help review my patch? Otherwise please skip this patch. Thanks!
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.002.patch, HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HDFS-3941:
-----------------------------
Target Version/s: 1.2.0 (was: 1.0.3)
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496621#comment-13496621 ]
Junping Du commented on HDFS-3941:
----------------------------------
Hi Nicholas. Ok. I will file a separated jira to fix the bug in trunk. Thanks for suggestions!
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498576#comment-13498576 ]
Junping Du commented on HDFS-3941:
----------------------------------
Hi, Jing, I am currently working on this. Due to time different (I am in +8), my response could be with latency. However, thanks for your patch.
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.002.patch, HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496612#comment-13496612 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3941:
----------------------------------------------
Hi Junping, the posted patch is quite different from the ones in HDFS-3498 and HDFS3601. It seems that there is some code refactoring. Could you make the patch look the same as the ones in trunk? For code refactoring, bug fixes or other additional works, let's do it separately so that the changes will also go to trunk.
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HDFS-3941:
-----------------------------
Attachment: HDFS-3941.patch
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du reassigned HDFS-3941:
--------------------------------
Assignee: Junping Du
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499431#comment-13499431 ]
Junping Du commented on HDFS-3941:
----------------------------------
Thanks Nicholas and Jing!
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Fix For: 1.2.0, 1-win
>
> Attachments: HDFS-3941.002.patch, HDFS-3941.003.patch, HDFS-3941.004.patch, HDFS-3941.005.patch, HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3941:
-----------------------------------------
Hadoop Flags: Reviewed
+1 the 005 patch looks good.
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.002.patch, HDFS-3941.003.patch, HDFS-3941.004.patch, HDFS-3941.005.patch, HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HDFS-3941:
-----------------------------
Description:
With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
0. No more than one replica is placed within a NodeGroup (*)
1. First replica on the local node.
2. Second and third replicas are within the same rack but remote rack with 1st replica.
3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
was:
With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
0. No more than one replica is placed within a NodeGroup (*)
1. First replica on the local node.
2. Second and third replicas are within the same rack but remote rack with 1st replica.
3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
Also, this patch abstract Replica Removal Policy
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao updated HDFS-3941:
----------------------------
Attachment: HDFS-3941.005.patch
Updated the patch to make it more compatible with the trunk. Have run unit tests locally and all the testcases passed except TestNNThroughputBenchmark (reported in HDFS-4204).
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.002.patch, HDFS-3941.003.patch, HDFS-3941.004.patch, HDFS-3941.005.patch, HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498147#comment-13498147 ]
Junping Du commented on HDFS-3941:
----------------------------------
Nicholas, I just file a related bug against trunk on HADOOP-9045. Can you take a look at it? Thanks! Hopefully, I can gradually eliminate gap between trunk and branch-1 patch without refactoring. However, I think bug fixing on branch-1 is what we should keep and try to update to trunk. Thoughts?
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498353#comment-13498353 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3941:
----------------------------------------------
> ... Can you take a look at it? ...
Sure, I have commented on HADOOP-9045.
We usually use a JIRA for a single issue/bug. The patch can be merged/backported to earlier branches within the same JIRA. In this case, let's keep HDFS-3941 for backporting HDFS-3498 and HDFS-3601, and then fix the bug by HADOOP-9045 (so we will commit HADOOP-9045 to both trunk and branch-1). Sounds good?
BTW, if you are busy on something, Jing can help out here.
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao updated HDFS-3941:
----------------------------
Attachment: HDFS-3941.004.patch
Change back the names of the three new methods in BlockPlacementPolicy to be compatible with the current trunk.
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.002.patch, HDFS-3941.003.patch, HDFS-3941.004.patch, HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HDFS-3941:
-----------------------------
Description:
With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
0. No more than one replica is placed within a NodeGroup (*)
1. First replica on the local node.
2. Second and third replicas are within the same rack but remote rack with 1st replica.
3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
Also, this patch abstract Replica Removal Policy
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498916#comment-13498916 ]
Junping Du commented on HDFS-3941:
----------------------------------
Jing, your patch seems to also include a bug fix which tries to update numOfAvailableNodes after removing same nodegroup nodes as replica placing candidates. A complete fix is patch available on HADOOP-9045. Per Nicholas's suggestion, we may only include code in HDFS-3498 and HDFS-3601?
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.002.patch, HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498549#comment-13498549 ]
Jing Zhao commented on HDFS-3941:
---------------------------------
Will run unit tests and testpatch tonight.
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HDFS-3941.002.patch, HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3941) Backport HDFS-3498 and HDFS3601:
update replica placement policy for new added "NodeGroup" layer topology
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE resolved HDFS-3941.
------------------------------------------
Resolution: Fixed
Fix Version/s: 1-win
1.2.0
I have committed it. Thanks, Junping and Jing!
> Backport HDFS-3498 and HDFS3601: update replica placement policy for new added "NodeGroup" layer topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3941
> URL: https://issues.apache.org/jira/browse/HDFS-3941
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Fix For: 1.2.0, 1-win
>
> Attachments: HDFS-3941.002.patch, HDFS-3941.003.patch, HDFS-3941.004.patch, HDFS-3941.005.patch, HDFS-3941.patch
>
>
> With enabling additional layer of "NodeGroup", the replica placement policy used in BlockPlacementPolicyWithNodeGroup is updated to following rules:
> 0. No more than one replica is placed within a NodeGroup (*)
> 1. First replica on the local node.
> 2. Second and third replicas are within the same rack but remote rack with 1st replica.
> 3. Other replicas on random nodes with restriction that no more than two replicas are placed in the same rack, if there is enough racks.
> Also, this patch abstract Replica Removal Policy from FSNameSystem to BlockPlacementPolicy and update removal policy slightly to remove duplicated replica within the same NodeGroup first when over-replicated happens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira