You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Junping Du (JIRA)" <ji...@apache.org> on 2012/09/16 20:44:07 UTC
[jira] [Created] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Junping Du created HADOOP-8820:
----------------------------------
Summary: Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
Key: HADOOP-8820
URL: https://issues.apache.org/jira/browse/HADOOP-8820
Project: Hadoop Common
Issue Type: New Feature
Components: net
Affects Versions: 1.0.0
Reporter: Junping Du
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496481#comment-13496481 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-8820:
------------------------------------------------
Junping, there is no easy way to review such a big patch in HADOOP-8817. It is simply too big. Also, branch-1 is a stable branch so that we must be conservative. Splitting it into 4 parts HADOOP-8820, HDFS-3941, HDFS-3942 and MAPREDUCE-4660 sounds really good. Would you mind doing it?
In the meantime, I will review your balancer patch for trunk.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496039#comment-13496039 ]
Junping Du commented on HADOOP-8820:
------------------------------------
Nicholas, Thanks for comments and questions. Under your great help, most HDFS part of code is committed into trunk (except Balancer), so the left parts missing in trunk are mostly YARN. HADOOP-8817 patch is based on MRV1 which has less code change comparing with YARN and MapReduce part is already reviewed (by Luke). Also, I did a lot of tests on a deployed virtualized hadoop cluster for combined patch, so I think it could be better to commit these patches as a whole patch to get rid of potential bug involved by sync and could be convenient for the patch to go to release branch (branch-1.2?). What do you think?
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao updated HADOOP-8820:
------------------------------
Attachment: HADOOP-8820.b1.003.patch
New patch based on HADOOP-8817 and Nicholas's comments. Will run ant test for the new patch.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8820:
-------------------------------
Attachment: HADOOP-8820.patch
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-8820:
------------------------------------
Status: Open (was: Patch Available)
Jenkins build does not pickup patch for 1.x. Canceling the patch.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496055#comment-13496055 ]
Jing Zhao commented on HADOOP-8820:
-----------------------------------
bq. The 1st is not a issue as children is already checked before.
I'm not sure if I understand the code correctly here. So the isRack() function first checks the children of the current node ("if (getChildren().isEmpty())"), but does not check the children of the first child of the current node. Thus is it possible that "((InnerNode) firstChild).children" is empty (in which case the following get(0) may cause an IndexOutofBoundsException)? Or somewhere outside the function it has been checked?
By the way, in the latest patch in HADOOP-8817 (HADOOP-8817-v3.patch), several method definitions seem to be repeated between BlockPlacementPolicy.java and BlockPlacementPolicyDefault.java (e.g., adjustSetsWithChosenReplica(), splitNodesWithLocalityGroup(), getLocalityGroupForSplit() and getRack()). So do we need to recheck the patch to verify?
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456711#comment-13456711 ]
Junping Du commented on HADOOP-8820:
------------------------------------
Thanks Suresh. Then, how does this patch marked as patch available for review?
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489896#comment-13489896 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-8820:
------------------------------------------------
> ... how does this patch marked as patch available for review?
You may simply post the patch on the JIRA.
The patch looks good. Some minor comments:
- The conf property net.topology.nodegroup.aware is not used. It is also not in the HADOOP-8469 and HADOOP-8470 patches. Let's add it in later patches.
- The netlock.writeLock().lock() below should be before try{}.
{code}
//NetworkTopologyWithNodeGroup.add(..)
+ try {
+ netlock.writeLock().lock();
+ Node rack = null;
{code}
Could you also run test-patch and all the unit tests with the new patch?
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496532#comment-13496532 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-8820:
------------------------------------------------
That's great. Thanks, Junping. Then, I will first commit HADOOP-8820.b1.002.patch. For the additional bug fixes, let's create a new JIRA since we also need to fix the code in trunk.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8820:
-------------------------------
Description:
This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
1. Make NetworkTopology class pluggable for extension.
2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496030#comment-13496030 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-8820:
------------------------------------------------
Junping, thanks for the update. I also want to commit the HADOOP-8817 patch directly. However, some parts of the patch is not yet committed to trunk. Your original plan is to have separated patches for various components. I think it is a good plan since it is easier to review the patches. Why don't we stick with it?
Jing, thanks for helping out here.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496570#comment-13496570 ]
Junping Du commented on HADOOP-8820:
------------------------------------
Great. Thanks Nicholas! Also, Thanks Jing for review and verification.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496093#comment-13496093 ]
Junping Du commented on HADOOP-8820:
------------------------------------
Hi,Jing. You are right that firstGrandChild is not checked before, I was thinking you are talking about firstChild. Also, the 4 methods you are listing here is just replicated. I already address your comments in v4 patch in HADOOP-8817. Thanks!
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495830#comment-13495830 ]
Junping Du commented on HADOOP-8820:
------------------------------------
Thanks Jing for the help here. However, I think the patch in HADOOP-8817 is the latest code and it fix a bug on placing replicas (the first repilca's host is not disallowed before). I did a lot of test (functional and performance) in a real deployed cluster on that patch, but my "ant test" still meet some trouble on my environment. Can you help to run ant test on that patch?
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8820:
-------------------------------
Target Version/s: 1.2.0 (was: 1.0.0)
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495839#comment-13495839 ]
Jing Zhao commented on HADOOP-8820:
-----------------------------------
Sure. I will re-generate the patch based on HADOOP-8817 and run ant test for the new patch. Can you give more details about the changes you made in 8817? Thanks.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495872#comment-13495872 ]
Junping Du commented on HADOOP-8820:
------------------------------------
On above two issues you point out, I think 2nd is potentially a issue and you can file a JIRA to follow up. The 1st is not a issue as children is already checked before.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao updated HADOOP-8820:
------------------------------
Attachment: HADOOP-8820.b1.002.patch
Hi Junping, maybe you're busy with other stuff. Since to address Nicholas's comments only needs simple changes, I just make these changes on your patch and run unit tests for you. All the test cases have passed on my machine. Thanks!
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496506#comment-13496506 ]
Junping Du commented on HADOOP-8820:
------------------------------------
Ok. Sounds good. I will split the patch into 4 patches. Thanks. Nicholas!
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495829#comment-13495829 ]
Jing Zhao commented on HADOOP-8820:
-----------------------------------
Besides, some other small issues:
1. {noformat}
+ @Override
+ boolean isRack() {
+ // it is node group
+ if (getChildren().isEmpty()) {
+ return false;
+ }
+
+ Node firstChild = children.get(0);
+
+ if (firstChild instanceof InnerNode) {
+ Node firstGrandChild = (((InnerNode) firstChild).children).get(0);
+ if (firstGrandChild instanceof InnerNode) {
{noformat}
I think here maybe firstChild.children can be empty thus the get(0) may cause an IndexOutofBoundsException. So maybe we need to check if children is empty first (and if it is, we can return true)?
2. {noformat}
+ public String getNodeGroup(String loc) {
+ try {
+ netlock.readLock().lock();
+ loc = InnerNode.normalize(loc);
+ Node locNode = getNode(loc);
+ if (locNode instanceof InnerNodeWithNodeGroup) {
+ InnerNodeWithNodeGroup node = (InnerNodeWithNodeGroup) locNode;
+ if (node.isNodeGroup()) {
+ return loc;
+ } else if (node.isRack()) {
+ // not sure the node group for a rack
+ return null;
+ } else {
+ // may be a leaf node
+ return getNodeGroup(node.getNetworkLocation());
+ }
{noformat}
Since the "locNode" is already an instance of InnerNodeWithNodeGroup, I think it cannot be a leaf node anymore, thus the code may never run into the "else" part?
I can create a separate jira to fix if you think the two issues stand.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496539#comment-13496539 ]
Jing Zhao commented on HADOOP-8820:
-----------------------------------
By the way, I've run unit tests locally for both 002 and 003 patches, and all the tests passed except TestFileCreation, which should be unrelated and has been reported in HDFS-4180.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492890#comment-13492890 ]
Junping Du commented on HADOOP-8820:
------------------------------------
Thanks Nicholas for reviewing. I will address your comments in HADOOP-8817 where I put all 4 patches together. Hopefully, it is easy to review and maintain.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated HADOOP-8820:
-------------------------------
Status: Patch Available (was: Open)
This patch is for branch-1 only.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495870#comment-13495870 ]
Junping Du commented on HADOOP-8820:
------------------------------------
Hi Jing, can you post "ant test" result on HADOOP-8817? I just adjust the patch due to Nicholas' comments.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495886#comment-13495886 ]
Junping Du commented on HADOOP-8820:
------------------------------------
Also, forget to note that, I think it could be convenient to have only one patch for "NodeGroup" layer with both HDFS and MAPREDUCE. So I suggest we don't commit this patch but commit HADOOP-8817 directly.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499798#comment-13499798 ]
Junping Du commented on HADOOP-8820:
------------------------------------
BTW, it is in HDFS-3495.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Fix For: 1.2.0, 1-win
>
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456639#comment-13456639 ]
Hadoop QA commented on HADOOP-8820:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12545333/HADOOP-8820.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1476//console
This message is automatically generated.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE resolved HADOOP-8820.
--------------------------------------------
Resolution: Fixed
Fix Version/s: 1-win
1.2.0
Hadoop Flags: Reviewed
+1 on HADOOP-8820.b1.002.patch
I have committed it. Thanks, Junping and Jing!
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Fix For: 1.2.0, 1-win
>
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du reassigned HADOOP-8820:
----------------------------------
Assignee: Junping Du
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499797#comment-13499797 ]
Junping Du commented on HADOOP-8820:
------------------------------------
Nicholas, I have updated balancer patch on trunk. Will you help to take a look at it? Thanks!
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Fix For: 1.2.0, 1-win
>
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and
HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as
NetworkTopologyWithNodeGroup)
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495863#comment-13495863 ]
Jing Zhao commented on HADOOP-8820:
-----------------------------------
Hi Junping, could you please recheck the new patch and make sure all the corresponding changes in HADOOP-8817 have been included? Thanks.
> Backport HADOOP-8469 and HADOOP-8470: add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-8820
> URL: https://issues.apache.org/jira/browse/HADOOP-8820
> Project: Hadoop Common
> Issue Type: New Feature
> Components: net
> Affects Versions: 1.0.0
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch
>
>
> This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
> 1. Make NetworkTopology class pluggable for extension.
> 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira