You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "stack (JIRA)" <ji...@apache.org> on 2007/09/20 21:00:35 UTC
[jira] Created: (HADOOP-1928) [hbase] Have master pass the
regionserver the filesystem to use
[hbase] Have master pass the regionserver the filesystem to use
---------------------------------------------------------------
Key: HADOOP-1928
URL: https://issues.apache.org/jira/browse/HADOOP-1928
Project: Hadoop
Issue Type: Improvement
Components: contrib/hbase
Reporter: stack
Priority: Minor
The less configuration needed starting a regionserver the better. Currently the regionserver needs to read hadoop-site mainly so it knows which filesystem to go against. If it could get this info from the master instead when it gets the response to its reporting-for-duty message, then the dependency on hadoop-site.xml could mostly go away.
Looking at regionserver, it looks like it would just take a little reorganization so that the hlog opening and the startup of worker threads doesn't happen until after we get our first response from the master.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1928) [hbase] Have master pass the
regionserver the filesystem to use
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-1928:
--------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Applied. Resolving.
> [hbase] Have master pass the regionserver the filesystem to use
> ---------------------------------------------------------------
>
> Key: HADOOP-1928
> URL: https://issues.apache.org/jira/browse/HADOOP-1928
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.15.0
>
> Attachments: master_passing_config-v2.patch, master_passing_config.patch
>
>
> The less configuration needed starting a regionserver the better. Currently the regionserver needs to read hadoop-site mainly so it knows which filesystem to go against. If it could get this info from the master instead when it gets the response to its reporting-for-duty message, then the dependency on hadoop-site.xml could mostly go away.
> Looking at regionserver, it looks like it would just take a little reorganization so that the hlog opening and the startup of worker threads doesn't happen until after we get our first response from the master.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1928) [hbase] Have master pass the
regionserver the filesystem to use
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-1928:
--------------------------
Attachment: master_passing_config.patch
Here's a first cut. Lots of HRegionServer refactoring so no fs access nor service thread startups until after we've had our first conversation with master. Needs testing on a cluster.
> [hbase] Have master pass the regionserver the filesystem to use
> ---------------------------------------------------------------
>
> Key: HADOOP-1928
> URL: https://issues.apache.org/jira/browse/HADOOP-1928
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Attachments: master_passing_config.patch
>
>
> The less configuration needed starting a regionserver the better. Currently the regionserver needs to read hadoop-site mainly so it knows which filesystem to go against. If it could get this info from the master instead when it gets the response to its reporting-for-duty message, then the dependency on hadoop-site.xml could mostly go away.
> Looking at regionserver, it looks like it would just take a little reorganization so that the hlog opening and the startup of worker threads doesn't happen until after we get our first response from the master.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1928) [hbase] Have master pass the
regionserver the filesystem to use
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529553 ]
stack commented on HADOOP-1928:
-------------------------------
Here is one possible approach:
+ On startup, a regionserver reports in for duty and usually, what happens next is that the master sends the regionserver a MSG_REGION_OPEN message with a payload of an HRegionInfo describing the particular region the master wants the regionserver to open. Other messages sent by the master include MSG_REGION_CLOSE, MSG_REGIONSERVER_STOP, etc.
+ Messages are comprised of a byte indicating the message type and optionally an HRegionInfo payload for a particular region if the message from the master addresses a particular region. Currently messages either have an empty payload or they carry an HRegionInfo payload (See HMsg).
+ We could add a new INIT message. The INIT message would be sent by the master the first time a regionserver reports in for duty. Its payload would be a MapWritable laden with properties the regionserver needs going about its work: e.g. filesystem to use and hbase.root to use, etc.
+ We'd change the HMsg so rather than an explicit HRegionInfo, instead it would carry a Writable whose type would be deteremined by the command carried in the message.
+ On the regionserver side, it would read the passed properties and set them into its local config. No attempts at touching the filesystem or startup of service threads would happen until after it had received the INIT message from the master.
> [hbase] Have master pass the regionserver the filesystem to use
> ---------------------------------------------------------------
>
> Key: HADOOP-1928
> URL: https://issues.apache.org/jira/browse/HADOOP-1928
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
>
> The less configuration needed starting a regionserver the better. Currently the regionserver needs to read hadoop-site mainly so it knows which filesystem to go against. If it could get this info from the master instead when it gets the response to its reporting-for-duty message, then the dependency on hadoop-site.xml could mostly go away.
> Looking at regionserver, it looks like it would just take a little reorganization so that the hlog opening and the startup of worker threads doesn't happen until after we get our first response from the master.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1928) [hbase] Have master pass the
regionserver the filesystem to use
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HADOOP-1928:
-----------------------------
Assignee: stack
> [hbase] Have master pass the regionserver the filesystem to use
> ---------------------------------------------------------------
>
> Key: HADOOP-1928
> URL: https://issues.apache.org/jira/browse/HADOOP-1928
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
>
> The less configuration needed starting a regionserver the better. Currently the regionserver needs to read hadoop-site mainly so it knows which filesystem to go against. If it could get this info from the master instead when it gets the response to its reporting-for-duty message, then the dependency on hadoop-site.xml could mostly go away.
> Looking at regionserver, it looks like it would just take a little reorganization so that the hlog opening and the startup of worker threads doesn't happen until after we get our first response from the master.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1928) [hbase] Have master pass the
regionserver the filesystem to use
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-1928:
--------------------------
Fix Version/s: 0.15.0
Status: Patch Available (was: Open)
Passes all local tests. Trying against hudson.
> [hbase] Have master pass the regionserver the filesystem to use
> ---------------------------------------------------------------
>
> Key: HADOOP-1928
> URL: https://issues.apache.org/jira/browse/HADOOP-1928
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.15.0
>
> Attachments: master_passing_config-v2.patch, master_passing_config.patch
>
>
> The less configuration needed starting a regionserver the better. Currently the regionserver needs to read hadoop-site mainly so it knows which filesystem to go against. If it could get this info from the master instead when it gets the response to its reporting-for-duty message, then the dependency on hadoop-site.xml could mostly go away.
> Looking at regionserver, it looks like it would just take a little reorganization so that the hlog opening and the startup of worker threads doesn't happen until after we get our first response from the master.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1928) [hbase] Have master pass the
regionserver the filesystem to use
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12530835 ]
Hadoop QA commented on HADOOP-1928:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12366601/master_passing_config-v2.patch
against trunk revision r579928.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/833/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/833/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/833/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/833/console
This message is automatically generated.
> [hbase] Have master pass the regionserver the filesystem to use
> ---------------------------------------------------------------
>
> Key: HADOOP-1928
> URL: https://issues.apache.org/jira/browse/HADOOP-1928
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.15.0
>
> Attachments: master_passing_config-v2.patch, master_passing_config.patch
>
>
> The less configuration needed starting a regionserver the better. Currently the regionserver needs to read hadoop-site mainly so it knows which filesystem to go against. If it could get this info from the master instead when it gets the response to its reporting-for-duty message, then the dependency on hadoop-site.xml could mostly go away.
> Looking at regionserver, it looks like it would just take a little reorganization so that the hlog opening and the startup of worker threads doesn't happen until after we get our first response from the master.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1928) [hbase] Have master pass the
regionserver the filesystem to use
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529958 ]
stack commented on HADOOP-1928:
-------------------------------
Chatting w/ Jim, rather than introduce a new INIT message, instead, change the signature on method the regionserver invokes to tell the master its reporting-for-duty. Currently it returns a void. Make it instead return a MapWritable laden w/ master-supplied configs.
> [hbase] Have master pass the regionserver the filesystem to use
> ---------------------------------------------------------------
>
> Key: HADOOP-1928
> URL: https://issues.apache.org/jira/browse/HADOOP-1928
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
>
> The less configuration needed starting a regionserver the better. Currently the regionserver needs to read hadoop-site mainly so it knows which filesystem to go against. If it could get this info from the master instead when it gets the response to its reporting-for-duty message, then the dependency on hadoop-site.xml could mostly go away.
> Looking at regionserver, it looks like it would just take a little reorganization so that the hlog opening and the startup of worker threads doesn't happen until after we get our first response from the master.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1928) [hbase] Have master pass the
regionserver the filesystem to use
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-1928:
--------------------------
Attachment: master_passing_config-v2.patch
v2 comes of edit and testing on cluster
HADOOP-1928 Have master pass the regionserver the filesystem to use
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestDFSAbort.java
Remove commented out lines (We used to have shorter than default
ipc timeouts and retries but was failing on hudson so went back to
using defaults again).
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/MultiRegionTable.java
Rootdir data member has been removed from HRegionServer. Use conf
setting.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/OOMERegionServer.java
HRegionServer constructor signature changed.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionServer.java
Refactor constructor. Delay filesystem access and service thread startups
until after first communique w/ master. Constructor no longer takes
the hbase.rootdir param. Instead, this is now read from the master
when regionserver reports in for duty. Made most of the data members
final.
(join): Refactored. Pass each thread to new join override.
(init): Added. Takes a MapWritable of configs obtained from master.
(startServiceThreads): Renamed from startAllServices.
(setupHlog, getThisIP): Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMasterRegionInterface.java
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
(regionServerStartup): Now returns subset of configuration to
regionserver.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/util/FSUtils.java
(isSystemAvaliable): Takes optional arg that can short-circuit retries
if another thread has ruled server should go down. Improved logging.
M src/contrib/hbase/build.xml
Add hbase conf dir to the test classpath. Means that hbase default
location will be under whatever has been specified as hadoop
default.
> [hbase] Have master pass the regionserver the filesystem to use
> ---------------------------------------------------------------
>
> Key: HADOOP-1928
> URL: https://issues.apache.org/jira/browse/HADOOP-1928
> Project: Hadoop
> Issue Type: Improvement
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.15.0
>
> Attachments: master_passing_config-v2.patch, master_passing_config.patch
>
>
> The less configuration needed starting a regionserver the better. Currently the regionserver needs to read hadoop-site mainly so it knows which filesystem to go against. If it could get this info from the master instead when it gets the response to its reporting-for-duty message, then the dependency on hadoop-site.xml could mostly go away.
> Looking at regionserver, it looks like it would just take a little reorganization so that the hlog opening and the startup of worker threads doesn't happen until after we get our first response from the master.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.