You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Junping Du (JIRA)" <ji...@apache.org> on 2012/06/04 07:40:22 UTC
[jira] [Created] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Junping Du created HADOOP-8472:
----------------------------------
Summary: Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Key: HADOOP-8472
URL: https://issues.apache.org/jira/browse/HADOOP-8472
Project: Hadoop Common
Issue Type: Sub-task
Components: ha, io
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Junping Du
Assignee: Junping Du
A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
1. The 3rd replica will be off node group of the 2nd replica
2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405659#comment-13405659 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-8472:
------------------------------------------------
> ... TopologyResolver is called in BlockPlacementPolicyWithNodeGroup ...
But all calls have withNodeGroupLayer == true so they can use the then-clause directly, e.g. in the getRack method below, use getFirstHalf(host.getNetworkLocation()) instead of getRack(host, true).
{code}
+ public static String getRack(Node host, boolean withNodeGroupLayer) {
+ return withNodeGroupLayer ? getFirstHalf(host.getNetworkLocation())
+ : host.getNetworkLocation();
+ }
{code}
The NodeGroup concept should only be visible in the pluggin. The original code should know nothing about it.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404675#comment-13404675 ]
Junping Du commented on HADOOP-8472:
------------------------------------
>From output message, test failed in TestFileAppend4 as cluster shutdown timeout which is not related to this patch. So restart the precommit test again.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8472:
-------------------------------
Release Note: This patches should be checked in together (or after) HDFS-3498 (was: This patches should be checked in together (or after) Hadoop-8471)
Status: Patch Available (was: Open)
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 2.0.0-alpha, 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403874#comment-13403874 ]
Hadoop QA commented on HADOOP-8472:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533970/HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.fs.viewfs.TestViewFsTrash
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1152//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1152//console
This message is automatically generated.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406788#comment-13406788 ]
Junping Du commented on HADOOP-8472:
------------------------------------
Looks like 7 core tests failure is irrelevant with this patch. I checked the other PreCommit build before/after this build, most with the same test failure. So I file a separated jira HADOOP-8557 to track this.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v5.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v6.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406357#comment-13406357 ]
Junping Du commented on HADOOP-8472:
------------------------------------
It looks like auto preCommit test is still down. Someone can help to manual start the preCommit test? Thanks!
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v5.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406592#comment-13406592 ]
Hadoop QA commented on HADOOP-8472:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12534761/HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v5.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.io.file.tfile.TestTFileByteArrays
org.apache.hadoop.io.file.tfile.TestTFileJClassComparatorByteArrays
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1166//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1166//console
This message is automatically generated.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v5.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v6.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8472:
-------------------------------
Status: Open (was: Patch Available)
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 2.0.0-alpha, 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405636#comment-13405636 ]
Junping Du commented on HADOOP-8472:
------------------------------------
As your previous comments, TopologyResolver is called in BlockPlacementPolicyWithNodeGroup which is in hdfs. If we don't like a separated class to handle this, how about reuse NetworkTopology to handle the methods that currently in TopologyResolver? I think it should be something in common.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405716#comment-13405716 ]
Junping Du commented on HADOOP-8472:
------------------------------------
Ok. got it. Will update code soon.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8472:
-------------------------------
Status: Open (was: Patch Available)
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 2.0.0-alpha, 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403161#comment-13403161 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-8472:
------------------------------------------------
If the refactoring is simple and easy, do it. Otherwise, we should focus on the main code but not waste our energy on the code style of the tests.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403831#comment-13403831 ]
Junping Du commented on HADOOP-8472:
------------------------------------
Hi Nicholas,
> If the refactoring is simple and easy, do it. Otherwise, we should focus on the main code but not waste our energy on the code style of the tests.
Agree. In fact, I try to refactor for some test cases but it looks odd to be a test code if we want to reuse everything in parent class (which means we will add a lot of hooks in parent testcases and override it in sub-class). I think current test implementation is explicit for reading and understanding the test flow. So in latest patch (v4), I only fix one java doc issue and do minimum refactoring on test.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404005#comment-13404005 ]
Hadoop QA commented on HADOOP-8472:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533989/HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestFileAppend4
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1155//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1155//console
This message is automatically generated.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405618#comment-13405618 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-8472:
------------------------------------------------
How about move the TopologyResolver methods to RackResolver since common and hdfs do not need them?
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405798#comment-13405798 ]
Junping Du commented on HADOOP-8472:
------------------------------------
@Nicholas, I already address your comments in new updated patch. Please help to review it again. Thanks!
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v5.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403139#comment-13403139 ]
Junping Du commented on HADOOP-8472:
------------------------------------
Honest saying, the test class TestReplicationPolicyWithNodeGroup get lots of code from TestReplicationPolicy but add test on NodeGroup relationship for each test scenario. Please comment if it is necessary to refactor the parent class (TestReplicationPolicy) then the sub-class can reuse most code. If so, I will do it soon.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8472:
-------------------------------
Attachment: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v6.patch
In v6 patch, add javadoc for BlockPlacementPolicyWithNodeGroup which is the only change.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v5.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v6.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8472:
-------------------------------
Attachment: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8472:
-------------------------------
Attachment: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch
As HADOOP-8471(HDFS-3498) is resolved and going to trunk, so here we can marked as patch available now.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405608#comment-13405608 ]
Junping Du commented on HADOOP-8472:
------------------------------------
Hey Nicholas,
Some mapreduce patch will use this method too. Please refer patch in MAPREDUCE-4310. Any suggestions?
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8472:
-------------------------------
Attachment: (was: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch)
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-8472:
-------------------------------------------
Issue Type: New Feature (was: Sub-task)
Parent: (was: HADOOP-8468)
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: New Feature
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v5.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v6.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406534#comment-13406534 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-8472:
------------------------------------------------
I just have started a build. The patch looks good to me except that please add some javadoc to BlockPlacementPolicyWithNodeGroup. Thanks.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v5.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8472:
-------------------------------
Release Note: This patches should be checked in together (or after) Hadoop-8471
Status: Patch Available (was: Open)
This patch have dependency on Hadoop-8471
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 2.0.0-alpha, 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8472:
-------------------------------
Attachment: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
This patch depends on HADOOP-8469(https://issues.apache.org/jira/browse/HADOOP-8469) and 8471(https://issues.apache.org/jira/browse/HADOOP-8471).
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403939#comment-13403939 ]
Junping Du commented on HADOOP-8472:
------------------------------------
The test error is at TestViewFsTrash again which is unrelated to the patch. I filed a separate jira to track this: HADOOP-8542, but cannot reproduce the test error on my local environment. I will restart the precommit test to see if error is still there.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-8472:
-------------------------------------------
Release Note: (was: This patches should be checked in together (or after) HDFS-3498)
Let move this issue to HDFS since BlockPlacementPolicy is in HDFS.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v5.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v6.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8472:
-------------------------------
Status: Patch Available (was: Open)
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 2.0.0-alpha, 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405596#comment-13405596 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-8472:
------------------------------------------------
Hi Junping,
It seems that all calls to TopologyResolver are in BlockPlacementPolicyWithNodeGroup and have withNodeGroupLayer == true. So is it possible to move the methods in TopologyResolver to BlockPlacementPolicyWithNodeGroup?
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8472:
-------------------------------
Attachment: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v5.patch
Two changes in v5 patch:
1. Remove TopologyResolver and TestTopologyResolver and put the static methods to NetworkTopology
2. Replace method of getRack(), getNodeGroup() with getFirstHalf() and getLastHalf() to hide NodeGroup info from pluggable class.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v5.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8472:
-------------------------------
Attachment: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403195#comment-13403195 ]
Hadoop QA commented on HADOOP-8472:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533832/HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1147//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1147//console
This message is automatically generated.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404693#comment-13404693 ]
Junping Du commented on HADOOP-8472:
------------------------------------
The failed test is tracked by a separated jira HDFS-3364 that filed before by Eli.
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8472) Implementation of
ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8472:
-------------------------------
Attachment: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch
> Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-8472
> URL: https://issues.apache.org/jira/browse/HADOOP-8472
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, io
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v2.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v3.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl-v4.patch, HADOOP-8472-BlockPlacementPolicyWithNodeGroup-impl.patch
>
>
> A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup was developed along with unit tests to support the four-layer hierarchical topology.
> The replica placement strategy used in ReplicaPlacementPolicyNodeGroup virtualization is almost the same as the original one. The differences are:
> 1. The 3rd replica will be off node group of the 2nd replica
> 2. If there is no local node available, the 1st replica will be placed on a node in the local node group.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira