You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "stack (JIRA)" <ji...@apache.org> on 2007/10/20 21:22:50 UTC
[jira] Created: (HADOOP-2084) [hbase] Add a LocalHBaseCluster
[hbase] Add a LocalHBaseCluster
-------------------------------
Key: HADOOP-2084
URL: https://issues.apache.org/jira/browse/HADOOP-2084
Project: Hadoop
Issue Type: New Feature
Reporter: stack
Priority: Minor
Make a MiniHBaseCluster-like class named LocalHBaseCluster. Key difference would be purge of MiniDFSCluster and FileSystem references/management and removal of utility that only makes sense in a unit testing context (Perhaps have MHBC inherit from LHBC). Make a new class rather than move MHBC from src/test to src/java to allow for later having master and regionserver do direct method invocation rather than invocation via RPC. LHRC would be used when hbase.master is set to the value 'local'.
LHRC would have a shutdown method to shutdown the cluster. MHBC brings up the cluster on construction. Perhaps LHRC should do different and not start the cluster until invocation of startup method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2084) [hbase] Add a LocalHBaseCluster
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2084:
--------------------------
Attachment: localhbasecluster-v3.patch
v3 includes benefit of a Jim review (Added some javadoc and removed unused imports).
> [hbase] Add a LocalHBaseCluster
> -------------------------------
>
> Key: HADOOP-2084
> URL: https://issues.apache.org/jira/browse/HADOOP-2084
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: localhbasecluster-v2.patch, localhbasecluster-v3.patch, localhbasecluster.patch
>
>
> Make a MiniHBaseCluster-like class named LocalHBaseCluster. Key difference would be purge of MiniDFSCluster and FileSystem references/management and removal of utility that only makes sense in a unit testing context (Perhaps have MHBC inherit from LHBC). Make a new class rather than move MHBC from src/test to src/java to allow for later having master and regionserver do direct method invocation rather than invocation via RPC. LHRC would be used when hbase.master is set to the value 'local'.
> LHRC would have a shutdown method to shutdown the cluster. MHBC brings up the cluster on construction. Perhaps LHRC should do different and not start the cluster until invocation of startup method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2084) [hbase] Add a LocalHBaseCluster
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2084:
--------------------------
Attachment: localhbasecluster-v2.patch
Previous patch had hadoop-2088 patch mistakenly mixed in.
> [hbase] Add a LocalHBaseCluster
> -------------------------------
>
> Key: HADOOP-2084
> URL: https://issues.apache.org/jira/browse/HADOOP-2084
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: localhbasecluster-v2.patch, localhbasecluster.patch
>
>
> Make a MiniHBaseCluster-like class named LocalHBaseCluster. Key difference would be purge of MiniDFSCluster and FileSystem references/management and removal of utility that only makes sense in a unit testing context (Perhaps have MHBC inherit from LHBC). Make a new class rather than move MHBC from src/test to src/java to allow for later having master and regionserver do direct method invocation rather than invocation via RPC. LHRC would be used when hbase.master is set to the value 'local'.
> LHRC would have a shutdown method to shutdown the cluster. MHBC brings up the cluster on construction. Perhaps LHRC should do different and not start the cluster until invocation of startup method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2084) [hbase] Add a LocalHBaseCluster
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2084:
--------------------------
Status: Patch Available (was: Open)
Passes locally. Trying hudson.
> [hbase] Add a LocalHBaseCluster
> -------------------------------
>
> Key: HADOOP-2084
> URL: https://issues.apache.org/jira/browse/HADOOP-2084
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: localhbasecluster-v2.patch, localhbasecluster.patch
>
>
> Make a MiniHBaseCluster-like class named LocalHBaseCluster. Key difference would be purge of MiniDFSCluster and FileSystem references/management and removal of utility that only makes sense in a unit testing context (Perhaps have MHBC inherit from LHBC). Make a new class rather than move MHBC from src/test to src/java to allow for later having master and regionserver do direct method invocation rather than invocation via RPC. LHRC would be used when hbase.master is set to the value 'local'.
> LHRC would have a shutdown method to shutdown the cluster. MHBC brings up the cluster on construction. Perhaps LHRC should do different and not start the cluster until invocation of startup method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2084) [hbase] Add a LocalHBaseCluster
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537493 ]
Hadoop QA commented on HADOOP-2084:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12368336/localhbasecluster-v3.patch
against trunk revision r588083.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/978/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/978/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/978/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/978/console
This message is automatically generated.
> [hbase] Add a LocalHBaseCluster
> -------------------------------
>
> Key: HADOOP-2084
> URL: https://issues.apache.org/jira/browse/HADOOP-2084
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: localhbasecluster-v2.patch, localhbasecluster-v3.patch, localhbasecluster.patch
>
>
> Make a MiniHBaseCluster-like class named LocalHBaseCluster. Key difference would be purge of MiniDFSCluster and FileSystem references/management and removal of utility that only makes sense in a unit testing context (Perhaps have MHBC inherit from LHBC). Make a new class rather than move MHBC from src/test to src/java to allow for later having master and regionserver do direct method invocation rather than invocation via RPC. LHRC would be used when hbase.master is set to the value 'local'.
> LHRC would have a shutdown method to shutdown the cluster. MHBC brings up the cluster on construction. Perhaps LHRC should do different and not start the cluster until invocation of startup method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2084) [hbase] Add a LocalHBaseCluster
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2084:
--------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed. Resolving.
> [hbase] Add a LocalHBaseCluster
> -------------------------------
>
> Key: HADOOP-2084
> URL: https://issues.apache.org/jira/browse/HADOOP-2084
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: localhbasecluster-v2.patch, localhbasecluster-v3.patch, localhbasecluster.patch
>
>
> Make a MiniHBaseCluster-like class named LocalHBaseCluster. Key difference would be purge of MiniDFSCluster and FileSystem references/management and removal of utility that only makes sense in a unit testing context (Perhaps have MHBC inherit from LHBC). Make a new class rather than move MHBC from src/test to src/java to allow for later having master and regionserver do direct method invocation rather than invocation via RPC. LHRC would be used when hbase.master is set to the value 'local'.
> LHRC would have a shutdown method to shutdown the cluster. MHBC brings up the cluster on construction. Perhaps LHRC should do different and not start the cluster until invocation of startup method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2084) [hbase] Add a LocalHBaseCluster
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2084:
--------------------------
Attachment: localhbasecluster.patch
Here's a patch w/ commit message below.
{code}
HADOOP-2084 Add a LocalHBaseCluster
Add new LocalHBaseCluster class, a derivative of the test class
MiniHBaseCluster. It runs a master and region server in same process.
Make this mini-cluster the default to mimic standalone mode in hadoop and nutch.
M src/contrib/hbase/conf/hbase-default.xml
Make default be 'local' rather than '0.0.0.0:60000'.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestInfoServers.java
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/HBaseClusterTestCase.java
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/MultiRegionTable.java
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestLogRolling.java
MiniHBaseCluster API changed.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/OOMEHMaster.java
HMaster doMain throws an exception now.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/MiniHBaseCluster.java
Delegate hbase cluster management to new LocalHBaseCluster.
Moved good bit of this classes' code up into new LHBC so available
beyond testing. Some cleanup while moving the code (no need of the
old MasterThread for instance).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionServer.java
Moved setup of RPC server into reportForDuty method from Constructor so could
construct a RegionServer without need of a master being up and running.
Also made a significant change in the way we go down. Master was just
waiting on the lease for all region servers to time out even if region
servers had actual gone down earlier. I (I believe it was I) had reasoned
that region server shouldn't send the EXIT message to the master if the
master had ordained the shutdown because master might not be there to
receive the shutdown... but master is waiting on all region server
leases to expire. If a region server does send the EXIT message,
the master will clean up its references and in general cluster shutdown
runs faster. Putting this facility back for now.
(doMain): Do not run a region server if hbase.master is 'local'.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HConstants.java
DEFAULT_MASTER_PORT: Added.
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/LocalHBaseCluster.java
Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
(doMain): Run LocalHBaseCluster if hbase.master is 'local'.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HConnectionManager.java
Handle 'local' mode.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HBaseConnection.java
Expose cloning.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/package.html
Update doc. to accomodate 'local' mode.
{code}
> [hbase] Add a LocalHBaseCluster
> -------------------------------
>
> Key: HADOOP-2084
> URL: https://issues.apache.org/jira/browse/HADOOP-2084
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: localhbasecluster.patch
>
>
> Make a MiniHBaseCluster-like class named LocalHBaseCluster. Key difference would be purge of MiniDFSCluster and FileSystem references/management and removal of utility that only makes sense in a unit testing context (Perhaps have MHBC inherit from LHBC). Make a new class rather than move MHBC from src/test to src/java to allow for later having master and regionserver do direct method invocation rather than invocation via RPC. LHRC would be used when hbase.master is set to the value 'local'.
> LHRC would have a shutdown method to shutdown the cluster. MHBC brings up the cluster on construction. Perhaps LHRC should do different and not start the cluster until invocation of startup method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2084) [hbase] Add a LocalHBaseCluster
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537686 ]
Hudson commented on HADOOP-2084:
--------------------------------
Integrated in Hadoop-Nightly #282 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/282/])
> [hbase] Add a LocalHBaseCluster
> -------------------------------
>
> Key: HADOOP-2084
> URL: https://issues.apache.org/jira/browse/HADOOP-2084
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
> Attachments: localhbasecluster-v2.patch, localhbasecluster-v3.patch, localhbasecluster.patch
>
>
> Make a MiniHBaseCluster-like class named LocalHBaseCluster. Key difference would be purge of MiniDFSCluster and FileSystem references/management and removal of utility that only makes sense in a unit testing context (Perhaps have MHBC inherit from LHBC). Make a new class rather than move MHBC from src/test to src/java to allow for later having master and regionserver do direct method invocation rather than invocation via RPC. LHRC would be used when hbase.master is set to the value 'local'.
> LHRC would have a shutdown method to shutdown the cluster. MHBC brings up the cluster on construction. Perhaps LHRC should do different and not start the cluster until invocation of startup method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2084) [hbase] Add a LocalHBaseCluster
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-2084:
--------------------------
Component/s: contrib/hbase
Fix Version/s: 0.16.0
Assignee: stack
> [hbase] Add a LocalHBaseCluster
> -------------------------------
>
> Key: HADOOP-2084
> URL: https://issues.apache.org/jira/browse/HADOOP-2084
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Reporter: stack
> Assignee: stack
> Priority: Minor
> Fix For: 0.16.0
>
>
> Make a MiniHBaseCluster-like class named LocalHBaseCluster. Key difference would be purge of MiniDFSCluster and FileSystem references/management and removal of utility that only makes sense in a unit testing context (Perhaps have MHBC inherit from LHBC). Make a new class rather than move MHBC from src/test to src/java to allow for later having master and regionserver do direct method invocation rather than invocation via RPC. LHRC would be used when hbase.master is set to the value 'local'.
> LHRC would have a shutdown method to shutdown the cluster. MHBC brings up the cluster on construction. Perhaps LHRC should do different and not start the cluster until invocation of startup method.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.