You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Created) (JIRA)" <ji...@apache.org> on 2012/02/18 14:44:59 UTC
[jira] [Created] (HADOOP-8091) Add a topology mapper that reads
hostname to rack mappings from a Java properties file
Add a topology mapper that reads hostname to rack mappings from a Java properties file
--------------------------------------------------------------------------------------
Key: HADOOP-8091
URL: https://issues.apache.org/jira/browse/HADOOP-8091
Project: Hadoop Common
Issue Type: New Feature
Components: util
Affects Versions: 0.24.0
Reporter: Steve Loughran
Priority: Minor
Getting mapping scripts right is surprisingly hard -and if its wrong in production bad things happen. It would be good to have something simpler for beginners -and one that is trivial to generate by a machine based on infrastructure data.
I propose adding an alternative mapper, one driven by a java property file
# the specific topology mapper must be identified for loading
# it uses another key to identify the property file to load. This is checked for on startup -if missing, fail.
# one property, perhaps "default-rack" identifies the default rack mapping for any host not in the list
# every other entry lists a hostname to rack mapping
# hostname mapping is done on the first entry in the FQDN, to be less brittle to domain resolution.
Example
{code}
default-rack=/rack1
host1=/rack1
host2=/rack1
host3=/rack2
host4=/rack2
{code}
Implementation
* add a new mapper that builds a concurrent hash map
* read in every entry in the specified property file, add it to the map
* when queried, extract the hostname (i.e. everything before any ".")
* match that in the hash table, return if found
* if not found: return the default rack
Feature creep would be to poll this file for changes at a (specified) frequency, and pick up the changes when they occur. This would require removing the caching topology mapper that wraps all others in the NN and RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8091) Add a topology mapper that reads
hostname to rack mappings from a Java properties file
Posted by "Steve Loughran (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-8091.
------------------------------------
Resolution: Duplicate
> Add a topology mapper that reads hostname to rack mappings from a Java properties file
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-8091
> URL: https://issues.apache.org/jira/browse/HADOOP-8091
> Project: Hadoop Common
> Issue Type: New Feature
> Components: util
> Affects Versions: 0.24.0
> Reporter: Steve Loughran
> Priority: Minor
>
> Getting mapping scripts right is surprisingly hard -and if its wrong in production bad things happen. It would be good to have something simpler for beginners -and one that is trivial to generate by a machine based on infrastructure data.
> I propose adding an alternative mapper, one driven by a java property file
> # the specific topology mapper must be identified for loading
> # it uses another key to identify the property file to load. This is checked for on startup -if missing, fail.
> # one property, perhaps "default-rack" identifies the default rack mapping for any host not in the list
> # every other entry lists a hostname to rack mapping
> # hostname mapping is done on the first entry in the FQDN, to be less brittle to domain resolution.
> Example
> {code}
> default-rack=/rack1
> host1=/rack1
> host2=/rack1
> host3=/rack2
> host4=/rack2
> {code}
> Implementation
> * add a new mapper that builds a concurrent hash map
> * read in every entry in the specified property file, add it to the map
> * when queried, extract the hostname (i.e. everything before any ".")
> * match that in the hash table, return if found
> * if not found: return the default rack
> Feature creep would be to poll this file for changes at a (specified) frequency, and pick up the changes when they occur. This would require removing the caching topology mapper that wraps all others in the NN and RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8091) Add a topology mapper that reads
hostname to rack mappings from a Java properties file
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13211437#comment-13211437 ]
Aaron T. Myers commented on HADOOP-8091:
----------------------------------------
Seems like this might be (very nearly) a duplicate of HADOOP-7030. Do you agree, Steve?
> Add a topology mapper that reads hostname to rack mappings from a Java properties file
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-8091
> URL: https://issues.apache.org/jira/browse/HADOOP-8091
> Project: Hadoop Common
> Issue Type: New Feature
> Components: util
> Affects Versions: 0.24.0
> Reporter: Steve Loughran
> Priority: Minor
>
> Getting mapping scripts right is surprisingly hard -and if its wrong in production bad things happen. It would be good to have something simpler for beginners -and one that is trivial to generate by a machine based on infrastructure data.
> I propose adding an alternative mapper, one driven by a java property file
> # the specific topology mapper must be identified for loading
> # it uses another key to identify the property file to load. This is checked for on startup -if missing, fail.
> # one property, perhaps "default-rack" identifies the default rack mapping for any host not in the list
> # every other entry lists a hostname to rack mapping
> # hostname mapping is done on the first entry in the FQDN, to be less brittle to domain resolution.
> Example
> {code}
> default-rack=/rack1
> host1=/rack1
> host2=/rack1
> host3=/rack2
> host4=/rack2
> {code}
> Implementation
> * add a new mapper that builds a concurrent hash map
> * read in every entry in the specified property file, add it to the map
> * when queried, extract the hostname (i.e. everything before any ".")
> * match that in the hash table, return if found
> * if not found: return the default rack
> Feature creep would be to poll this file for changes at a (specified) frequency, and pick up the changes when they occur. This would require removing the caching topology mapper that wraps all others in the NN and RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8091) Add a topology mapper that reads
hostname to rack mappings from a Java properties file
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213715#comment-13213715 ]
Steve Loughran commented on HADOOP-8091:
----------------------------------------
HADOOP-7030 is almost superset; so yes. The nice thing about java properties is they have a standard format for encoding non-ascii stuff and comments, and are fairly well known in the java land (as well as coming with a built in Properties class), so are very easy to do.
> Add a topology mapper that reads hostname to rack mappings from a Java properties file
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-8091
> URL: https://issues.apache.org/jira/browse/HADOOP-8091
> Project: Hadoop Common
> Issue Type: New Feature
> Components: util
> Affects Versions: 0.24.0
> Reporter: Steve Loughran
> Priority: Minor
>
> Getting mapping scripts right is surprisingly hard -and if its wrong in production bad things happen. It would be good to have something simpler for beginners -and one that is trivial to generate by a machine based on infrastructure data.
> I propose adding an alternative mapper, one driven by a java property file
> # the specific topology mapper must be identified for loading
> # it uses another key to identify the property file to load. This is checked for on startup -if missing, fail.
> # one property, perhaps "default-rack" identifies the default rack mapping for any host not in the list
> # every other entry lists a hostname to rack mapping
> # hostname mapping is done on the first entry in the FQDN, to be less brittle to domain resolution.
> Example
> {code}
> default-rack=/rack1
> host1=/rack1
> host2=/rack1
> host3=/rack2
> host4=/rack2
> {code}
> Implementation
> * add a new mapper that builds a concurrent hash map
> * read in every entry in the specified property file, add it to the map
> * when queried, extract the hostname (i.e. everything before any ".")
> * match that in the hash table, return if found
> * if not found: return the default rack
> Feature creep would be to poll this file for changes at a (specified) frequency, and pick up the changes when they occur. This would require removing the caching topology mapper that wraps all others in the NN and RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira