You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Bryan Duxbury (JIRA)" <ji...@apache.org> on 2008/02/18 22:16:34 UTC
[jira] Created: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Factor Master into Master, RegionManager, and ServerManager
-----------------------------------------------------------
Key: HBASE-457
URL: https://issues.apache.org/jira/browse/HBASE-457
Project: Hadoop HBase
Issue Type: Sub-task
Reporter: Bryan Duxbury
Assignee: Bryan Duxbury
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HBASE-457:
--------------------------------
Attachment: 457-v4.patch
This patch makes it so the vast majority of fields on RegionManager are no longer used directly.
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Attachments: 457-v2.patch, 457-v3.patch, 457-v4.patch, 457.patch
>
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HBASE-457:
--------------------------------
Status: Patch Available (was: In Progress)
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Attachments: 457.patch
>
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12571795#action_12571795 ]
stack commented on HBASE-457:
-----------------------------
In the below in metascanner...
{code}
+ /** Work for the meta scanner is queued up here */
+ private volatile BlockingQueue<MetaRegion> metaRegionsToScan =
+ new LinkedBlockingQueue<MetaRegion>();
{code}
...change the volatile to final. I think your hope was that the volatile would apply to the content of metaRegionsToScan. Rather it applies to the metaRegionsToScan reference... and that ain't going to change once its been allocated (See http://java.sun.com/docs/books/jls/third_edition/html/classes.html#8.3.1.4).
You should change the few like the below in ServerManager also:
{code}
+ /** The map of known server names to server info */
+ protected volatile Map<String, HServerInfo> serversToServerInfo =
+ new ConcurrentHashMap<String, HServerInfo>();
{code}
Do data members in ServerManager have to be protected? Can they not be private or are they need to be protected for tests? E.g:
{code}
+ protected HMaster master;
+
+ protected final Leases serverLeases;
{code}
If need to be protected for tests then lets make an issue to fix this using something like this, http://www.onjava.com/pub/a/onjava/2003/11/12/reflection.html, if its not too much of a pain.
Does only the master have a regionManager? If so, why not master.getRegionManager and make the method package-private? Then it would not have to be passed to the MetaScanner constructor? Not important -- passing on construction saves an indirection every time... just a thought... would help w/ the below from ProcessServerShutdown:
{code}
- master.onlineMetaRegions.remove(info.getStartKey());
+ master.regionManager.offlineMetaRegion(info.getStartKey());
{code}
I like the cleanup done in the below:
{code}
- ArrayList<MetaRegion> regions = new ArrayList<MetaRegion>();
- synchronized (master.onlineMetaRegions) {
- regions.addAll(master.onlineMetaRegions.values());
- }
+ List<MetaRegion> regions = regionManager.getListOfOnlineMetaRegions();
{code}
(Not important), the below is a TODO?
{code}
+ /** compute the average load across all region servers */
+ public int averageLoad() {
+ return 0;
+ }
{code}
I like the new setUnassigned and noLongerPending, getRootRegionLocation, etc., methods
Fix the volatiles in RegionManager... make them finals. You can do this now you declare and define all in the one statement.
I ran the patch locally and it passed. Suggest that you address what you think important in the above -- at least the volatiles I'd say -- and if tests pass locally for you, just apply v5.
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Attachments: 457-v2.patch, 457-v3.patch, 457-v4.patch, 457.patch
>
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12571210#action_12571210 ]
stack commented on HBASE-457:
-----------------------------
Here's a few comments on the patch:
+ This seems odd in MetaScanner: 'synchronized (master.regionManager.metaScannerLock) ' Would think classes should never have this kind of intimate knowledge of others: i.e. MetaScanner knowing that master has a data member named regionManager which in turn has an object to lock. Or this kinda thing: 'master.regionManager.onlineMetaRegions' or 'master.regionManager.metaRegionsToScan.pol' (I understand this is first stage of an extensive refactoring and that these once-inner classes are tough to unravel but maybe something can be done about this intimacy?). This problem seems prevalent throughout the patch.
+ Can these changes:
{code}
- master.onlineMetaRegions.remove(info.getStartKey());
+ master.regionManager.onlineMetaRegions.remove(info.getStartKey());
{code}
be replace with a method on master named something like removeOnlineMetaRegion(final Text startKey)?
One suggestion would be to make a getRegionManager method for the master (Are these supposed to be singletons?) and then a removeOnlineMetaRegions method on RegionManager.
Same for these in ServerManager:
{code}
+ HServerAddress root = master.getRootRegionLocation();
+ if (root != null && root.equals(storedInfo.getServerAddress())) {
+ master.regionManager.unassignRootRegion();
+ }
+ master.delayedToDoQueue.put(new ProcessServerShutdown(master, storedInfo));
{code}
+ I like the ServerManager and RegionManager classes
+ Can ServerManager and RegionManager be finals on master? Can they be private? (smile)
+ Just remove this kinda stuff rather than leave it commented in:
{code}
+/* synchronized (master.regionManager.onlineMetaRegions) {
+ metaRegion = master.regionManager.onlineMetaRegions.size() == 1 ?
+ master.regionManager.onlineMetaRegions.get(master.regionManager.onlineMetaRegions.firstKey()) :
+ master.regionManager.onlineMetaRegions.containsKey(regionInfo.getRegionName()) ?
+ master.regionManager.onlineMetaRegions.get(regionInfo.getRegionName()) :
+ master.regionManager.onlineMetaRegions.get(master.regionManager.onlineMetaRegions.headMap(
regionInfo.getRegionName()).lastKey());
- }
- this.metaRegionName = metaRegion.getRegionName();
+ }*/
{code}
There is another example of above in SortedSoftMap.
Otherwise, patch is definetly going in right direction.
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Attachments: 457-v2.patch, 457.patch
>
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HBASE-457:
--------------------------------
Attachment: 457-v2.patch
Here's another version. It's re-merged with trunk.
It also now includes a few small changes to SoftSortedMap and HConnectionManager to try and deal with the random failures we've been seeing of NPEs in getRegionLocation. The (non-reproducible) theory is that between checking if the cache for a table is empty and actually getting the key, it gets garbage collected into emptiness.
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Attachments: 457-v2.patch, 457.patch
>
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HBASE-457:
--------------------------------
Component/s: master
Priority: Minor (was: Major)
Description:
Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
ServerManager keeps track of servers - leases, message processing, load and load average, etc.
RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12570559#action_12570559 ]
Jim Kellerman commented on HBASE-457:
-------------------------------------
+0 TestTableMapReduce failed with:
junit.framework.AssertionFailedError
at org.apache.hadoop.hbase.MultiRegionTable.makeMultiRegionTable(MultiRegionTable.java:150)
at org.apache.hadoop.hbase.mapred.TestTableMapReduce.localTestMultiRegionTable(TestTableMapReduce.java:279)
at org.apache.hadoop.hbase.mapred.TestTableMapReduce.testTableMapReduce(TestTableMapReduce.java:206)
On the other hand TestTableIndex which runs the same code, passed.
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Attachments: 457-v2.patch, 457.patch
>
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-457:
--------------------------------
Fix Version/s: 0.2.0
Affects Version/s: 0.2.0
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Affects Versions: 0.2.0
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Fix For: 0.2.0
>
> Attachments: 457-v2.patch, 457-v3.patch, 457-v4.patch, 457.patch
>
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12571190#action_12571190 ]
Bryan Duxbury commented on HBASE-457:
-------------------------------------
That error in MultiRegionTable is just checking if the split parent region's directory still exists. If it doesn't exist, doesn't that mean that the daughters could have been compacted and therefore the parent would have been cleaned up? How could this be an issue caused by master refactoring?
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Attachments: 457-v2.patch, 457.patch
>
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12571585#action_12571585 ]
stack commented on HBASE-457:
-----------------------------
Is this change intentional where you use regionManager instead of master in the below?
{code}
- while (!master.closed.get() && !master.rootScanned &&
- master.rootRegionLocation.get() == null) {
+ while (!master.closed.get() && !regionManager.rootScanned &&
+ regionManager.rootRegionLocation.get() == null) {
{code}
These are unfortunate in BaseScanner but shouldn't block the patch: 'synchronized (regionManager.killList) {' i.e. synchronizing on a lock that is over in another class. We can work on them over time.
Should we be passing in a ServerManager into Base/MetaScanner on construction? Would it help w/ the likes of the below:
{code}
- storedInfo = master.serversToServerInfo.get(serverName);
- deadServer = master.deadServers.contains(serverName);
+ storedInfo = master.serverManager.serversToServerInfo.get(serverName);
+ deadServer = master.serverManager.deadServers.contains(serverName);
{code}
Would suggest adding a getServerManager accessor on master otherwise.
I like these changes:
{code}
- master.onlineMetaRegions.put(region.getStartKey(), region);
+ regionManager.putMetaRegionOnline(region);
{code}
This code seems odd:
{code}
- master.initialMetaScanComplete = true;
+ regionManager.initialMetaScanComplete = true;
{code}
Why does the region manager have the flag for whether a scan is complete -- meta or root? Could this be kept in the scanner classes themselves?
You could also add a setter to regionManager for this kinda stuff (so MetaScanner doesn't have to know about regionmanager data members)
Should you be passing RegionManager to TableOperation constructor so you don't have to do this?
{code}
- master.onlineMetaRegions.remove(info.getStartKey());
+ master.regionManager.offlineMetaRegion(info.getStartKey());
{code}
Should below just be changed to a master.getRegionLocation?
{code}
- server = master.connection.getHRegionConnection(master.rootRegionLocation.get());
+ server = master.connection.getHRegionConnection(
+ master.regionManager.rootRegionLocation.get());
{code}
In ServerManager you have this:
{code}
+ protected volatile Set<String> deadServers =
+ Collections.synchronizedSet(new HashSet<String>());
{code}
Would suggest the volatile be changed to a final and that you default data members as private till some one complains they need it otherwise (Same for other members of this class)
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Attachments: 457-v2.patch, 457-v3.patch, 457.patch
>
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HBASE-457:
--------------------------------
Attachment: 457.patch
Here's my first draft.
I believe that all the state is segmented how it should be, but the methods of access are pretty muddy still. For instance, I'd greatly prefer it if external classes didn't have to directly access the various region state lists, but since they have to synchronize on them, it's tricky.
Also, ServerManager and RegionManager are coupled tightly right now, as ServerManager passes messages to RegionManager pretty much explicitly. It'd be preferable to allow any class to subscribe to certain message types from ServerManager, and then we could add other cool consumers of the event data.
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Attachments: 457.patch
>
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HBASE-457:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Attachments: 457-v2.patch, 457-v3.patch, 457-v4.patch, 457.patch
>
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12570572#action_12570572 ]
Bryan Duxbury commented on HBASE-457:
-------------------------------------
Do you consistently get that failure? What does the failure mean? (We should at least change the failure to include a message, don't you think?)
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Attachments: 457-v2.patch, 457.patch
>
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated HBASE-457:
--------------------------------
Attachment: 457-v3.patch
Here's a version with some of the problems fixed. This isn't the final product, but it is an incremental step in that direction.
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Components: master
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
> Priority: Minor
> Attachments: 457-v2.patch, 457-v3.patch, 457.patch
>
>
> Even with TableOperation and descendants factored out of HMaster, it's still a huge class. Every bit of master state basically lives in one class, so it's very challenging to understand the logical groupings of everything.
> To make this a little more manageable, let's make two new abstractions, ServerManager and RegionManager.
> ServerManager keeps track of servers - leases, message processing, load and load average, etc.
> RegionManager keeps track of the root location, meta table online state, assigning regions to servers, and so on.
> HMaster then keeps around one of each of these classes to track state and do it's master stuff. The HMaster class itself does not change it's interface to external consumers. It also retains the main processing loop, HBase closed state, start and stop, etc.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (HBASE-457) Factor Master into Master,
RegionManager, and ServerManager
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HBASE-457 started by Bryan Duxbury.
> Factor Master into Master, RegionManager, and ServerManager
> -----------------------------------------------------------
>
> Key: HBASE-457
> URL: https://issues.apache.org/jira/browse/HBASE-457
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Bryan Duxbury
> Assignee: Bryan Duxbury
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.