You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Patrick Angeles (JIRA)" <ji...@apache.org> on 2010/11/12 06:09:13 UTC
[jira] Created: (HADOOP-7030) new topology mapping implementations
new topology mapping implementations
------------------------------------
Key: HADOOP-7030
URL: https://issues.apache.org/jira/browse/HADOOP-7030
Project: Hadoop Common
Issue Type: New Feature
Affects Versions: 0.21.0, 0.20.2, 0.20.1
Reporter: Patrick Angeles
Attachments: topology.patch
The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501515#comment-13501515 ]
Matt Foley commented on HADOOP-7030:
------------------------------------
moved target version to 1.2.0 upon publishing 1.1.1 RC.
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 2.0.0-alpha
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030-branch-1.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237535#comment-13237535 ]
Hudson commented on HADOOP-7030:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk #1029 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1029/])
HADOOP-7030. Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597)
Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 0.23.3
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030-branch-1.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237518#comment-13237518 ]
Hudson commented on HADOOP-7030:
--------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #235 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/235/])
Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599)
Result = FAILURE
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 0.23.3
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030-branch-1.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7030) new topology mapping
implementations
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931329#action_12931329 ]
Hadoop QA commented on HADOOP-7030:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12459424/topology.patch
against trunk revision 1034184.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 16 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 4 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/95//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/95//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/95//console
This message is automatically generated.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Attachments: topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7030:
------------------------------
Resolution: Fixed
Fix Version/s: 0.23.3
Target Version/s: 1.1.0, 0.23.3 (was: 0.23.3, 1.1.0)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Patrick!
I opened HADOOP-8204 for the test failure, which was unrelated to the change.
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 0.23.3
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284630#comment-13284630 ]
Matt Foley edited comment on HADOOP-7030 at 5/29/12 6:38 AM:
-------------------------------------------------------------
Patch not committed in time for 1.1.0.
was (Author: mattf):
Patch not submitted in time for 1.1.0.
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 2.0.0-alpha
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030-branch-1.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-7030:
-------------------------------
Target Version/s: 0.23.3, 1.2.0 (was: 1.1.1, 0.23.3)
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 2.0.0-alpha
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030-branch-1.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7030) new topology mapping
implementations
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996873#comment-12996873 ]
Hadoop QA commented on HADOOP-7030:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12469308/HADOOP-7030-2.patch
against trunk revision 1071364.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 26 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/252//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/252//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/252//console
This message is automatically generated.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237213#comment-13237213 ]
Hudson commented on HADOOP-7030:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1931 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1931/])
HADOOP-7030. Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597)
Result = ABORTED
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 0.23.3
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237154#comment-13237154 ]
Hudson commented on HADOOP-7030:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1996 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1996/])
HADOOP-7030. Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597)
Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 0.23.3
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-7030) new topology mapping
implementations
Posted by "Tom White (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White reassigned HADOOP-7030:
---------------------------------
Assignee: Tom White (was: Patrick Angeles)
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) new topology mapping
implementations
Posted by "Patrick Angeles (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235253#comment-13235253 ]
Patrick Angeles commented on HADOOP-7030:
-----------------------------------------
+1 on the patch.
Thanks for picking this one up, Tom!
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7030:
------------------------------
Target Version/s: 1.1.0, 0.23.3 (was: 0.23.3, 1.1.0)
Summary: Add TableMapping topology implementation to read host to rack mapping from a file (was: new topology mapping implementations)
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237227#comment-13237227 ]
Hudson commented on HADOOP-7030:
--------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #729 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/729/])
Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599)
Result = ABORTED
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 0.23.3
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237508#comment-13237508 ]
Hudson commented on HADOOP-7030:
--------------------------------
Integrated in Hadoop-Hdfs-trunk #994 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/994/])
HADOOP-7030. Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597)
Result = FAILURE
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 0.23.3
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030-branch-1.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-7030) new topology mapping implementations
Posted by "Patrick Angeles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Angeles updated HADOOP-7030:
------------------------------------
Attachment: HADOOP-7030-2.patch
Updated patch that introduces the RefreshableDNSToSwitchMapping interface which adds a 'refresh' method to DNSToSwitchMapping. This can be used later as a hook for 'hadoop (dfs|mr)admin -refreshNodes' to update the topology without restarting the cluster.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HADOOP-7030) new topology mapping
implementations
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235866#comment-13235866 ]
Steve Loughran commented on HADOOP-7030:
----------------------------------------
#the accessors are good for testing.
#I've been slowly putting the changes into the code, but gradually and trying to avoid any form of problem w/existing custom topologies.
What I do want to do -and they arent a jiras, I should add them are
* A command line entry point to take a list of hostnames, look at the current config, tell you what they map to and list the final map. This lets you preflight check the operations.
* A way of getting the dump of the topo map in use by a service, such as through one of the JSP pages. That way people can see what's gone wrong.
Being able to get into the map is what you need for these.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) new topology mapping
implementations
Posted by "Tom White (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235873#comment-13235873 ]
Tom White commented on HADOOP-7030:
-----------------------------------
Steve - these are good features, but it's better to introduce API changes at the time the feature is added so that you and reviewers are sure the API fits the use case.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7030) new topology mapping
implementations
Posted by "Patrick Angeles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002252#comment-13002252 ]
Patrick Angeles commented on HADOOP-7030:
-----------------------------------------
- Could you combine the two types of file, so that if there are three columns the first two are interpreted as a range, otherwise use the first as a single host. Or just support CIDR notation?
- I'd prefer to keep them separate as the first two columns have completely different meanings when using one style (table lookup) over the other (IP-range).
BTW, I don't think CIDR is appropriate here. For the table-based mapping, you can get either hosts or IPs, possibly depending on who (JT or NN) is requesting the rack ID. The docs are unclear here.
In either case, servers within a rack rarely fall in units that are powers of two.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Patrick Angeles
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) new topology mapping
implementations
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236026#comment-13236026 ]
Alejandro Abdelnur commented on HADOOP-7030:
--------------------------------------------
+1 after a few minor corrections:
1. Reader creation simplification:
{code}
File file = new File(filename);
reader = new BufferedReader(new InputStreamReader(new FileInputStream(
file)));
{code}
should be:
{code}
File file = new File(filename);
reader = new BufferedReader(new FileReader(filename));
{code}
2. non-comment & invalid lines (not 2 columns) should write a log warning instead failing silently.
3. rename refresh() to load() or initialize(). refresh() gives the impression of well refresh() and I'm not sure this is the intention.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7030) new topology mapping
implementations
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986585#action_12986585 ]
Hadoop QA commented on HADOOP-7030:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12469308/HADOOP-7030-2.patch
against trunk revision 1063090.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 26 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/197//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/197//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/197//console
This message is automatically generated.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-7030) new topology mapping
implementations
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931342#action_12931342 ]
Hadoop QA commented on HADOOP-7030:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12459427/HADOOP-7030.patch
against trunk revision 1034184.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 16 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/96//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/96//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/96//console
This message is automatically generated.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Attachments: HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7030) new topology mapping implementations
Posted by "Patrick Angeles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Angeles updated HADOOP-7030:
------------------------------------
Attachment: HADOOP-7030.patch
Reworked to fix 3 of the 4 findbugs issues. 4th one is harmless.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Attachments: HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7030:
------------------------------
Attachment: HADOOP-7030-branch-1.patch
Attaching a backport for branch-1.
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 0.23.3
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030-branch-1.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7030) new topology mapping implementations
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7030:
------------------------------
Attachment: HADOOP-7030.patch
Updated patch to use CachedDNSToSwitchMapping.
bq. on .23+ it will automatically pick up any extra features put into the base class I added. E.g permitting providing info to callers about whether the config is multi-switch, allowing callers to get/dump the topology map for diagnostics
As far as I can tell nothing calls these new methods. We don't normally add new methods unless they are used.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-7030) new topology mapping implementations
Posted by "Patrick Angeles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Angeles updated HADOOP-7030:
------------------------------------
Status: Patch Available (was: Open)
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.21.0, 0.20.2, 0.20.1
> Reporter: Patrick Angeles
> Attachments: topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-7030:
-------------------------------
Target Version/s: 1.1.1, 0.23.3 (was: 1.1.0, 0.23.3)
Patch not submitted in time for 1.1.0.
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 2.0.0-alpha
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030-branch-1.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7030) new topology mapping implementations
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7030:
------------------------------
Target Version/s: 1.1.0, 0.23.3
Status: Patch Available (was: Open)
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.21.0, 0.20.2, 0.20.1
> Reporter: Patrick Angeles
> Assignee: Tom White
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237137#comment-13237137 ]
Hudson commented on HADOOP-7030:
--------------------------------
Integrated in Hadoop-Common-0.23-Commit #721 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/721/])
Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599)
Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 0.23.3
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-7030) new topology mapping implementations
Posted by "Patrick Angeles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Angeles updated HADOOP-7030:
------------------------------------
Attachment: topology.patch
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Attachments: topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-7030) new topology mapping implementations
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7030:
------------------------------
Assignee: Patrick Angeles
Status: Open (was: Patch Available)
This looks like a useful addition. Here are my comments on the patch:
* Could you combine the two types of file, so that if there are three columns the first two are interpreted as a range, otherwise use the first as a single host. Or just support CIDR notation?
* Have you thought about InetAddress to avoid implementing IP address parsing logic? http://guava-libraries.googlecode.com/svn/tags/release08/javadoc/com/google/common/net/InetAddresses.html might be useful (there was talk of introducing Guava recently).
* RefreshableDNSToSwitchMapping isn't hooked up yet, so perhaps it should go in a follow on JIRA.
* The name "TableMapping" is a bit general. How about "FileBasedMapping", or similar?
* The configuration keys should go in CommonConfigurationKeysPublic.
* Primes are not needed in hashCode implementations. For Ip4 Arrays.hashCode(value) is sufficient.
* The tests swallow exceptions - there should at least be a comment saying that this is expected. Also, fail() with a message is preferable to assertTrue(false).
* The tests should be JUnit 4 style.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.21.0, 0.20.2, 0.20.1
> Reporter: Patrick Angeles
> Assignee: Patrick Angeles
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7030) new topology mapping implementations
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7030:
------------------------------
Attachment: HADOOP-7030.patch
New patch addressing Alejandro's feedback.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237513#comment-13237513 ]
Hudson commented on HADOOP-7030:
--------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #207 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/207/])
Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599)
Result = UNSTABLE
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 0.23.3
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030-branch-1.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7030) new topology mapping
implementations
Posted by "Patrick Angeles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002039#comment-13002039 ]
Patrick Angeles commented on HADOOP-7030:
-----------------------------------------
Hey Tom,
Thanks for the review. Here are some responses:
Could you combine the two types of file, so that if there are three columns the first two are interpreted as a range, otherwise use the first as a single host. Or just support CIDR notation?
- I'd prefer to keep them separate as the first two columns have completely different meanings when using one style (table lookup) over the other (IP-range).
Have you thought about InetAddress to avoid implementing IP address parsing logic? http://guava-libraries.googlecode.com/svn/tags/release08/javadoc/com/google/common/net/InetAddresses.html might be useful (there was talk of introducing Guava recently).
- I have not. Will look into this, although I'd rather keep this patch lightweight and not require the addition of another jar.
RefreshableDNSToSwitchMapping isn't hooked up yet, so perhaps it should go in a follow on JIRA.
- Yes, that is the intent. Those JIRAs would go into MAPREDUCE and HDFS.
The name "TableMapping" is a bit general. How about "FileBasedMapping", or similar?
- I'm willing to listen to suggestions, however I think FileBasedMapping is even more vague :)
The configuration keys should go in CommonConfigurationKeysPublic.
- Since this is a pluggable interface, I should not have to modify existing core code. That's better WRT separation of concerns and componentization. I'm willing to take your suggestion if the general consensus is that I should :)
Primes are not needed in hashCode implementations. For Ip4 Arrays.hashCode(value) is sufficient.
- Ok.
The tests swallow exceptions - there should at least be a comment saying that this is expected. Also, fail() with a message is preferable to assertTrue(false).
- Ok.
The tests should be JUnit 4 style.
- Ok.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Patrick Angeles
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) new topology mapping
implementations
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235486#comment-13235486 ]
Steve Loughran commented on HADOOP-7030:
----------------------------------------
Tom, could you have the class extend the cached mapping, the way the scripted one does?
This would mean that
# the code to manage hash tables are in there
# on .23+ it will automatically pick up any extra features put into the base class I added. E.g permitting providing info to callers about whether the config is multi-switch, allowing callers to get/dump the topology map for diagnostics, etc -yet the same code will work for both branches, and be a better starting point for adding refresh operations.
# those bits of the code that check for the mapping being a cached mapping won't wrap the mapping in another cache, so making it easier to do updates &C. As you note, that bit of {{RackResolver}} should be fixed at some point -but subclassing the cached mapper would do this automatically.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) new topology mapping
implementations
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235867#comment-13235867 ]
Hadoop QA commented on HADOOP-7030:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519484/HADOOP-7030.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/745//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/745//console
This message is automatically generated.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Kang Xiao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285326#comment-13285326 ]
Kang Xiao commented on HADOOP-7030:
-----------------------------------
tow question about the patch
1.If new rack and host added to the cluster, it will resolve rack name to DEFAULT_RACK. It maybe better to allow reload of the config file.
2.The following code set initialized to be true before load(). It may cause the config file not fully loaded if some exception throwed by load().
{code}
+ public synchronized List<String> resolve(List<String> names) {
+ if (!initialized) {
+ initialized = true;
+ load();
+ }
{code}
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 2.0.0-alpha
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030-branch-1.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7030) new topology mapping implementations
Posted by "Tom White (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-7030:
------------------------------
Attachment: HADOOP-7030.patch
I've created a simplified patch which only has the two-column layout (we can add the range variant later as needed - also, now we have Guava the implementation will be simplified since we won't need to model IP addresses). I've also removed the refreshable interface, since as Steve notes it doesn't work well with the way that resolvers currently use the mapping interface (that should be revisited too).
In addition to the unit tests provided by Patrick, I tested on a small cluster and observed that the rack mappings I specified were picked up correctly.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Patrick Angeles
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237149#comment-13237149 ]
Hudson commented on HADOOP-7030:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #1922 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1922/])
HADOOP-7030. Add TableMapping topology implementation to read host to rack mapping from a file. Contributed by Patrick Angeles and tomwhite. (Revision 1304597)
Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304597
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 0.23.3
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424232#comment-13424232 ]
Steve Loughran commented on HADOOP-7030:
----------------------------------------
1. -conf file reload is a bigger issue/need than just this file, as it really requires cache reset and remote calls, but I've had a thought there specifically for the use case of "new host added". Filed as HADOOP-8629
2: filed as HADOOP-8628.
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 2.0.0-alpha
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030-branch-1.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) Add TableMapping topology
implementation to read host to rack mapping from a file
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237130#comment-13237130 ]
Hudson commented on HADOOP-7030:
--------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #711 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/711/])
Merge -r 1304596:1304597 from trunk to branch-0.23. Fixes: HADOOP-7030 (Revision 1304599)
Result = SUCCESS
tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1304599
Files :
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/TableMapping.java
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/net/TestTableMapping.java
> Add TableMapping topology implementation to read host to rack mapping from a file
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Fix For: 0.23.3
>
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) new topology mapping
implementations
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236960#comment-13236960 ]
Hadoop QA commented on HADOOP-7030:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519688/HADOOP-7030.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.ha.TestHealthMonitor
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/756//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/756//console
This message is automatically generated.
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Tom White
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7030) new topology mapping
implementations
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213779#comment-13213779 ]
Steve Loughran commented on HADOOP-7030:
----------------------------------------
With the changes I've been doing to the topology classes (with an optional abstract base class), these classes would be best if
{{RefreshableDNSToSwitchMapping}} was a subclass of the new {{AbstractDNSToSwitchMapping}}.
On barrier to any dynamic/refreshable topology class is that the values get cached. In particular
{{org.apache.hadoop.yarn.util.RackResolver}} checks to see if the mapping is an instance of {{CachedDNSToSwitchMapping}} -and if not, wraps it in one. the DNManager doesn't do anything so dramatic - it does do an instance check, but only to trigger a pre-emptive lookup of all known hosts.
Unless we add flags to {{AbstractDNSToSwitchMapping}} to provide more info about the resolver (e.g. caching, dynamic, etc), it's probably safest to subclass {{CachedDNSToSwitchMapping}} and overwrite its resolve method (and anything else that is useful).
> new topology mapping implementations
> ------------------------------------
>
> Key: HADOOP-7030
> URL: https://issues.apache.org/jira/browse/HADOOP-7030
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.20.1, 0.20.2, 0.21.0
> Reporter: Patrick Angeles
> Assignee: Patrick Angeles
> Attachments: HADOOP-7030-2.patch, HADOOP-7030.patch, topology.patch
>
>
> The default ScriptBasedMapping implementation of DNSToSwitchMapping for determining cluster topology has some drawbacks. Principally, it forks to an OS-specific script.
> This issue proposes two new Java implementations of DNSToSwitchMapping. TableMapping reads a two column text file that maps an IP or hostname to a rack ID. Ip4RangeMapping reads a three column text file where each line represents a start and end IP range plus a rack ID.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira