You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Created) (JIRA)" <ji...@apache.org> on 2012/03/30 18:36:31 UTC
[jira] [Created] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Provide a command line entry point to view/test topology options
----------------------------------------------------------------
Key: HADOOP-8232
URL: https://issues.apache.org/jira/browse/HADOOP-8232
Project: Hadoop Common
Issue Type: Sub-task
Components: util
Affects Versions: 0.23.1
Reporter: Steve Loughran
Priority: Minor
Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
Target audience:
# ops teams trying to get a new/changed script working before deploying it on a cluster.
# someone trying to write their first script.
Resolve and list the rack mappings of the given host
{code}
hadoop topo test [host1] [host2] ...
{code}
This would load the hostnames from a given file, resolve all of them and list the results:
{code}
hadoop topo testfile filename
{code}
This version is intended for the ops team who have a list of hostnames, IP addresses.
* Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
* No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242555#comment-13242555 ]
Steve Loughran commented on HADOOP-8232:
----------------------------------------
Test output when a configuration is loaded that lists a map class that does not load
{code}
bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/bin/hadoop --config /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/invalid-mapper org.apache.hadoop.net.TopologyTool testfile /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/hosts.txt
net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ThisMappingClassIsNotDefined
2012-03-30 18:12:56,225 ERROR net.TopologyTool (TopologyTool.java:run(77)) - Failed to load the DNS mapping org.apache.hadoop.net.ThisMappingClassIsNotDefined: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.net.ThisMappingClassIsNotDefined not found
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.net.ThisMappingClassIsNotDefined not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1298)
at org.apache.hadoop.net.AbstractDNSToSwitchMapping.createCachingDNSToSwitchMapping(AbstractDNSToSwitchMapping.java:166)
at org.apache.hadoop.net.TopologyTool.run(TopologyTool.java:73)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.net.TopologyTool.main(TopologyTool.java:228)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.net.ThisMappingClassIsNotDefined not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1266)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1290)
... 5 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.net.ThisMappingClassIsNotDefined not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1175)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1264)
... 6 more
2012-03-30 18:12:56,229 ERROR net.TopologyTool (TopologyTool.java:run(79)) - The configuration option is wrong, or the classpath is incomplete
{code}
The error message attempts to hint at the likely causes.
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252638#comment-13252638 ]
Hadoop QA commented on HADOOP-8232:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522448/HADOOP-8232.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/849//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/849//console
This message is automatically generated.
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch, HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242560#comment-13242560 ]
Steve Loughran commented on HADOOP-8232:
----------------------------------------
Output when a script is provided that does not exist
{code}
bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/bin/hadoop --config /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/missing-script org.apache.hadoop.net.TopologyTool test single-host
net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping
Caching wrapper class = class org.apache.hadoop.net.ScriptBasedMapping
Mapping script filename= "there-is-no-script-with-this-name.py"
Mapping script path = "/Users/slo/Java/Hadoop/github/trunk/there-is-no-script-with-this-name.py"
2012-03-30 18:16:04,211 WARN net.TopologyTool (TopologyTool.java:run(91)) - Script file not found -the script must be in the execution path
Instance information: script-based mapping with script "there-is-no-script-with-this-name.py"
Topology is not considered single-switch
Resolving single-host
2012-03-30 18:16:04,346 WARN net.ScriptBasedMapping (ScriptBasedMapping.java:runResolveCommand(245)) - Exception running there-is-no-script-with-this-name.py single-host
java.io.IOException: Cannot run program "there-is-no-script-with-this-name.py" (in directory "/Users/slo/Java/Hadoop/github/trunk"): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:206)
at org.apache.hadoop.util.Shell.run(Shell.java:188)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381)
at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:242)
at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:180)
at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
at org.apache.hadoop.net.TopologyTool.resolveHostnames(TopologyTool.java:189)
at org.apache.hadoop.net.TopologyTool.run(TopologyTool.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.net.TopologyTool.main(TopologyTool.java:228)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:53)
at java.lang.ProcessImpl.start(ProcessImpl.java:91)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 11 more
2012-03-30 18:16:04,352 WARN net.TopologyTool (TopologyTool.java:resolveHostnames(192)) - Hostname resolution returned a null list
Final topology:
Mapping: script-based mapping with script "there-is-no-script-with-this-name.py"
Known mappings:
Nodes: 0
Switches: 0
{code}
Here the exec() stack trace is being caught and printed inside the {{ScriptBasedMapping}} class -all the {{TopologyTool}} sees is a null value being passed back from the resolve operation.
After execution the map is dumped and shown to be empty.
It's hard to give better diagnostics on the failure when the failure itself isn't visible to the caller. At least it is now visible outside the daemons.
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242565#comment-13242565 ]
Steve Loughran commented on HADOOP-8232:
----------------------------------------
Test with table mapping pointing to a table that does not exist relative to the directory where the hadoop command is being invoked from:
{code}
bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/bin/hadoop --config /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/missing-table org.apache.hadoop.net.TopologyTool test 4 home extra
net.topology.node.switch.mapping.impl=org.apache.hadoop.net.TableMapping
Caching wrapper class = class org.apache.hadoop.net.TableMapping
Instance information: cached switch mapping relaying to TableMapping with table "table.txt"
Path: /Users/slo/Java/Hadoop/github/trunk/table.txt
Map size: 0
Topology is not considered single-switch
Resolving 4
2012-03-30 18:22:03,831 WARN net.TableMapping (TableMapping.java:load(113)) - table.txt cannot be read. /default-rack will be returned.
java.io.FileNotFoundException: table.txt (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at java.io.FileInputStream.<init>(FileInputStream.java:79)
at java.io.FileReader.<init>(FileReader.java:41)
at org.apache.hadoop.net.TableMapping$RawTableMapping.load(TableMapping.java:98)
at org.apache.hadoop.net.TableMapping$RawTableMapping.resolve(TableMapping.java:132)
at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
at org.apache.hadoop.net.TopologyTool.resolveHostnames(TopologyTool.java:189)
at org.apache.hadoop.net.TopologyTool.run(TopologyTool.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.net.TopologyTool.main(TopologyTool.java:228)
Hostname "4" resolved to "/default-rack" in 15.548 milliseconds
Resolving home
Hostname "home" resolved to "/default-rack" in 6.812 milliseconds
Resolving extra
Hostname "extra" resolved to "/default-rack" in 0.702 milliseconds
Final topology:
Mapping: cached switch mapping relaying to TableMapping with table "table.txt"
Path: /Users/slo/Java/Hadoop/github/trunk/table.txt
Map size: 0
Known mappings:
home -> /default-rack
extra -> /default-rack
4 -> /default-rack
Nodes: 3
Switches: 1
{code}
The error message is printed inside the TableMapping, and is not visible to the caller. It could be improved by listing the absolute path to the table.
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran reassigned HADOOP-8232:
--------------------------------------
Assignee: Steve Loughran
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HADOOP-8232) Provide a command line entry
point to view/test topology options
Posted by "Steve Loughran (Work started) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HADOOP-8232 started by Steve Loughran.
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-8232:
-----------------------------------
Attachment: HADOOP-8232.patch
#table mapping dumps its state, or the exception raised when trying to load it
#topology dump warns if there is >1 switch and the list includes a /default-rack entry, as that is a common sign of incomplete configuration.
No tests yet, not ready to go in.
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch, HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-8232:
-----------------------------------
Attachment: HADOOP-8232.patch
Does not include tests or patch bin/hadoop
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242553#comment-13242553 ]
Steve Loughran commented on HADOOP-8232:
----------------------------------------
Testing with the {{echo}} script.
{code}
bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/bin/hadoop --config /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/valid-script org.apache.hadoop.net.TopologyTool testfile /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/hosts.txt
net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping
Caching wrapper class = class org.apache.hadoop.net.ScriptBasedMapping
Mapping script filename= "echo"
Mapping script path = "/Users/slo/Java/Hadoop/github/trunk/echo"
2012-03-30 18:03:37,291 WARN net.TopologyTool (TopologyTool.java:run(91)) - Script file not found -the script must be in the execution path
Instance information: script-based mapping with script "echo"
Topology is not considered single-switch
Resolving host1
Hostname "host1" resolved to "host1" in 24.4 milliseconds
Resolving host2
Hostname "host2" resolved to "host2" in 12.681 milliseconds
Resolving host3
Hostname "host3" resolved to "host3" in 9.753 milliseconds
Resolving host4
Hostname "host4" resolved to "host4" in 10.084 milliseconds
Resolving host5
Hostname "host5" resolved to "host5" in 9.431 milliseconds
Resolving host5
Hostname "host5" resolved to "host5" in 0.128 milliseconds
Resolving host3
Hostname "host3" resolved to "host3" in 0.056 milliseconds
Resolving host1.hadoop.apache.org
Hostname "host1.hadoop.apache.org" resolved to "host1.hadoop.apache.org" in 8.377 milliseconds
Final topology:
Mapping: script-based mapping with script "echo"
Known mappings:
host1.hadoop.apache.org -> host1.hadoop.apache.org
host5 -> host5
host4 -> host4
host3 -> host3
host2 -> host2
host1 -> host1
Nodes: 6
Switches: 6
{code}
# the entry point first lists the mapping class and creates it.
# the statement {{Caching wrapper class = class org.apache.hadoop.net.ScriptBasedMapping}} means that the script has not been wrapped by a CachingDNS mapper, as it was not deemed necessary (maybe explain that better in the output?)
# the script filename is extracted from the configuration and printed
# the absolute path to the script is resolved -to avoid working directory confusion.
# because there is no file at that point the program warns that the file must be on the path (maybe: not use log.warn)
# the {{mapping.toString()}} operation is called to see what the string has to say about itself, I've been slowly making those useful while still robust.
# the topology is asked whether or not it considers itself single switch. As there is a non-null script, it is assumed to be multi-switch.
# the hosts are resolved with their times.
# when a cached hostname is resolved execution time drops significantly, showing caching works
# finally, the topology is dumped, which for the caching DNS mapper means a dump of its (hostname, switch) map.
# from the topology dump the number of hosts and switches is counted.
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242563#comment-13242563 ]
Steve Loughran commented on HADOOP-8232:
----------------------------------------
Table mapping without any map file provided
{code}
bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/bin/hadoop --config /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/undefined-table org.apache.hadoop.net.TopologyTool test 4 home extra
net.topology.node.switch.mapping.impl=org.apache.hadoop.net.TableMapping
Caching wrapper class = class org.apache.hadoop.net.TableMapping
Instance information: cached switch mapping relaying to TableMapping with table ""
Map size: 0
Topology is not considered single-switch
Resolving 4
2012-03-30 18:19:28,114 WARN net.TableMapping (TableMapping.java:load(91)) - net.topology.table.file.name not configured. /default-rack will be returned.
Hostname "4" resolved to "/default-rack" in 30.982 milliseconds
Resolving home
Hostname "home" resolved to "/default-rack" in 15.165 milliseconds
Resolving extra
Hostname "extra" resolved to "/default-rack" in 0.591 milliseconds
Final topology:
Mapping: cached switch mapping relaying to TableMapping with table ""
Map size: 0
Known mappings:
home -> /default-rack
extra -> /default-rack
4 -> /default-rack
Nodes: 3
Switches: 1
{code}
The names resolve -the delay on the first operation is because that is when table loading is attempted.
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242544#comment-13242544 ]
Steve Loughran commented on HADOOP-8232:
----------------------------------------
I'm not sure the best way to automate this; I think isolating the inner code from the entry point, testing that -including by capturing the output- would be the best tactic.
Manual testing so far is done with some different configurations
[https://github.com/steveloughran/hadoop-patches/tree/master/misc/topology]
#script based mapper with a script that is not on the path
#a script mapper where the script is echo
#a table mapper with no table defined
#a table mapper with a table that is defined but missing
#a valid table mapper
Tests with hostnames in a file and on the command line were tried; examples will be shown.
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-8232:
-----------------------------------
Status: Patch Available (was: In Progress)
for jenkins only
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242558#comment-13242558 ]
Hadoop QA commented on HADOOP-8232:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12520625/HADOOP-8232.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/803//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/803//console
This message is automatically generated.
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-8232:
-----------------------------------
Status: Patch Available (was: Open)
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch, HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242566#comment-13242566 ]
Steve Loughran commented on HADOOP-8232:
----------------------------------------
Test with table mapping pointing to a table that does not exist relative to the directory where the hadoop command is being invoked from:
{code}
bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/bin/hadoop --config /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/missing-table org.apache.hadoop.net.TopologyTool test 4 home extra
net.topology.node.switch.mapping.impl=org.apache.hadoop.net.TableMapping
Caching wrapper class = class org.apache.hadoop.net.TableMapping
Instance information: cached switch mapping relaying to TableMapping with table "table.txt"
Path: /Users/slo/Java/Hadoop/github/trunk/table.txt
Map size: 0
Topology is not considered single-switch
Resolving 4
2012-03-30 18:22:03,831 WARN net.TableMapping (TableMapping.java:load(113)) - table.txt cannot be read. /default-rack will be returned.
java.io.FileNotFoundException: table.txt (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at java.io.FileInputStream.<init>(FileInputStream.java:79)
at java.io.FileReader.<init>(FileReader.java:41)
at org.apache.hadoop.net.TableMapping$RawTableMapping.load(TableMapping.java:98)
at org.apache.hadoop.net.TableMapping$RawTableMapping.resolve(TableMapping.java:132)
at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
at org.apache.hadoop.net.TopologyTool.resolveHostnames(TopologyTool.java:189)
at org.apache.hadoop.net.TopologyTool.run(TopologyTool.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.net.TopologyTool.main(TopologyTool.java:228)
Hostname "4" resolved to "/default-rack" in 15.548 milliseconds
Resolving home
Hostname "home" resolved to "/default-rack" in 6.812 milliseconds
Resolving extra
Hostname "extra" resolved to "/default-rack" in 0.702 milliseconds
Final topology:
Mapping: cached switch mapping relaying to TableMapping with table "table.txt"
Path: /Users/slo/Java/Hadoop/github/trunk/table.txt
Map size: 0
Known mappings:
home -> /default-rack
extra -> /default-rack
4 -> /default-rack
Nodes: 3
Switches: 1
{code}
The error message is printed inside the TableMapping, and is not visible to the caller. It could be improved by listing the absolute path to the table.
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242574#comment-13242574 ]
Steve Loughran commented on HADOOP-8232:
----------------------------------------
test with a table that can be found; the table is
{code}
host1 /rack1
host2 /rack1
host3 /rack2/switch1
host4 /rack2/switch1
{code}
the output is
{code}
bash hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/bin/hadoop --config /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table org.apache.hadoop.net.TopologyTool testfile /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/hosts.txt
net.topology.node.switch.mapping.impl=org.apache.hadoop.net.TableMapping
Caching wrapper class = class org.apache.hadoop.net.TableMapping
Instance information: cached switch mapping relaying to TableMapping with table "/Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table/table.txt"
Path: /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table/table.txt
Map size: 0
Topology is not considered single-switch
Resolving host1
Hostname "host1" resolved to "/rack1" in 17.476 milliseconds
Resolving host2
Hostname "host2" resolved to "/rack1" in 0.878 milliseconds
Resolving host3
Hostname "host3" resolved to "/rack2/switch1" in 0.428 milliseconds
Resolving host4
Hostname "host4" resolved to "/rack2/switch1" in 0.34 milliseconds
Resolving host5
Hostname "host5" resolved to "/default-rack" in 0.578 milliseconds
Resolving host5
Hostname "host5" resolved to "/default-rack" in 0.095 milliseconds
Resolving host3
Hostname "host3" resolved to "/rack2/switch1" in 0.023 milliseconds
Resolving host1.hadoop.apache.org
Hostname "host1.hadoop.apache.org" resolved to "/default-rack" in 0.555 milliseconds
Final topology:
Mapping: cached switch mapping relaying to TableMapping with table "/Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table/table.txt"
Path: /Users/slo/Java/Hadoop/github/hadoop-patches/misc/topology/table/table.txt
Map size: 4
Known mappings:
host1.hadoop.apache.org -> /default-rack
host5 -> /default-rack
host4 -> /rack2/switch1
host3 -> /rack2/switch1
host2 -> /rack1
host1 -> /rack1
Nodes: 6
Switches: 3
{code}
The output at the end is less than ideal. The "Map Size" value is from the {{TableMapping.toString()}} method, which, in the patch, lists it. The name could be changed, but what it represents can't: it is the size of the hostname->rack values provided by the table file.
The map that is then printed below it is the map known by the CachedDNSSwitchMapping, which implements the method needed to get a copy of its mapping for diagnostics purposes.
Ideally, both should be shown, which could be done by having the inner {{RawTableMapping}} extend {{AbstractDNSToSwitchMapping}} and implement the {{getSwitchMap()}} method by returning its static map. That still needs to be made visible somehow through {{CachedDNSMapping}}
An alternative would just be for {{TableMapping.getSwitchMap()} to merge its cached values with those provided by the original table, so displaying everything that came in the table, and any cached values for unknown hosts now mapped to /default-rack.
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8232) Provide a command line entry point
to view/test topology options
Posted by "Steve Loughran (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-8232:
-----------------------------------
Status: Open (was: Patch Available)
> Provide a command line entry point to view/test topology options
> ----------------------------------------------------------------
>
> Key: HADOOP-8232
> URL: https://issues.apache.org/jira/browse/HADOOP-8232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: util
> Affects Versions: 0.23.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-8232.patch
>
>
> Add a new command line entry point "topo" with commands for preflight checking of a clusters topology setup.
> The initial operations would be to list the implementation class of the mapper, and attempt to load it, resolve a set of supplied hostnames, then dump the topology map after the resolution process.
> Target audience:
> # ops teams trying to get a new/changed script working before deploying it on a cluster.
> # someone trying to write their first script.
> Resolve and list the rack mappings of the given host
> {code}
> hadoop topo test [host1] [host2] ...
> {code}
> This would load the hostnames from a given file, resolve all of them and list the results:
> {code}
> hadoop topo testfile filename
> {code}
> This version is intended for the ops team who have a list of hostnames, IP addresses.
> * Rather than just list them, the ops team may want to mandate that there were no /default-rack mappings found, as that is invariably a sign that the script isn't handling a hostname properly.
> * No attempt to be clever and do IP address resolution, FQDN to hostname mapping, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira