You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2009/09/04 01:45:57 UTC
[jira] Created: (HBASE-1816) Master rewrite
Master rewrite
--------------
Key: HBASE-1816
URL: https://issues.apache.org/jira/browse/HBASE-1816
Project: Hadoop HBase
Issue Type: Task
Components: master
Reporter: stack
Priority: Blocker
Fix For: 0.21.0
Attachments: HBase Hackathon Notes - Sunday.pdf
An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1816) Master rewrite
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771288#action_12771288 ]
stack commented on HBASE-1816:
------------------------------
Removed hbase-1062 band-aid (hope its ok Andrew). Was getting in way of tests that used to wait around till safe-mode was cleared... but safe-mode has been removed. Test would put up blocker and wait on compactions before proceeding. But compactions wouldn't compact enough until the compaction thread had turned on compactions fully.. which would take a while. By the time they were fully operational, test had timed out. hbase-1062 was a band-aid. IIRC, conversation w/ apurtell had it that we should remove it now anyways since we no longer compact on open -- only if references (which is not usual on open).
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf, master-v2.patch, master-v3.patch, master-v4.patch, master.patch, meta-cleanup.patch
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1816) Master rewrite
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1816:
-------------------------
Attachment: master.patch
A start. Clean up of main and start on data members and constructor.
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf, master.patch, meta-cleanup.patch
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1816) Master rewrite
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771447#action_12771447 ]
Andrew Purtell commented on HBASE-1816:
---------------------------------------
+1 HBASE-1062 band aid removal.
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf, master-v2.patch, master-v3.patch, master-v4.patch, master.patch, meta-cleanup.patch
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1816) Master rewrite
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1816:
-------------------------
Attachment: master-v3.patch
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf, master-v2.patch, master-v3.patch, master.patch, meta-cleanup.patch
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1816) Master rewrite
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12754627#action_12754627 ]
stack commented on HBASE-1816:
------------------------------
Started page up in wiki: http://wiki.apache.org/hadoop/Hbase/MasterRewrite. Will try and write up a bit of design for new server here.
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf, meta-cleanup.patch
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1816) Master rewrite
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770703#action_12770703 ]
stack commented on HBASE-1816:
------------------------------
Committed v4 above.
Does following:
+ Downs the retries when tests are running from 10 to 4.
+ Made the RegionManager and ServerManager public (but kept their constructors package protected) so can live outside Master; made it so could get rid of a bunch of redirection where methods were duplicated in Master and all they did was go to RegionManager.
+ Removed unused data members and accessors in HMaster
+ Made all in HMaster private. Added accessors. Removed all mentions of ServerManager from RegionManager but unable to do the reciprocal just yet. There is really ugly stuff where in ServerManager, we'll sync on the RegionManager and then do a bunch of RegionManager operations. Hopefully most of these go when move table state and schema to zk.
+ Moved HMsg defines from ServerManager to HMsg where they belong
+ Cleaned up cruddy javadoc
+ Move fs methods out of HMaster to FSUtils.
+ Some cleanup of zk watchers. Moved method from HMaster out to zk master watcher class
HMaster still too big.
Working on moving state and schema to zk now. Will do as per the design where I write a JSON file that has an entry per table with state and schema
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf, master-v2.patch, master-v3.patch, master-v4.patch, master.patch, meta-cleanup.patch
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1816) Master rewrite
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771169#action_12771169 ]
stack commented on HBASE-1816:
------------------------------
Removed master lease period from conf and handling in RS and master. Its no longer needed now we have zk.
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf, master-v2.patch, master-v3.patch, master-v4.patch, master.patch, meta-cleanup.patch
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1816) Master rewrite
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770804#action_12770804 ]
stack commented on HBASE-1816:
------------------------------
Removed safe-mode.
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf, master-v2.patch, master-v3.patch, master-v4.patch, master.patch, meta-cleanup.patch
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1816) Master rewrite
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769508#action_12769508 ]
stack commented on HBASE-1816:
------------------------------
I've been adding to http://wiki.apache.org/hadoop/Hbase/MasterRewrite. Have proposal for moving schema and state out to zk but still a bit stuck on how to do regions. Regions need to stay in .META. once they've been deployed, I think, because .META. scales better and zk (though patrick and mohadev yesterday claimed 100k small objects should be fine in a dedicated quorum) and because you can't do getClosest against zk -- not yet anyways.
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf, master.patch, meta-cleanup.patch
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1816) Master rewrite
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1816:
-------------------------
Attachment: HBase Hackathon Notes - Sunday.pdf
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1816) Master rewrite
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1816:
-------------------------
Attachment: master-v4.patch
This patch I'll commit in morning if all tests pass. It doesn't add much. It removes some unused stuff from HMaster, moves some of its stuff out to FSUtils, cleans up some of the methods, makes all private and adds accessors rather than let package members have direct access to HMaster internals. I then went through RegionManager and removed all ServerManager references. I couldn't do vice-versa. This stuff is a real tangle.
After committing above, I'm going to start work on moving schema and table state out to zk. That'll decimate this package undoing the need for a lot of whats in here.
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf, master-v2.patch, master-v3.patch, master-v4.patch, master.patch, meta-cleanup.patch
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1816) Master rewrite
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-1816:
--------------------------------------
Attachment: meta-cleanup.patch
Patch that removes a lot of meta and root stuff that's not needed when .META. is in ZK. It does not compile.
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf, meta-cleanup.patch
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1816) Master rewrite
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779207#action_12779207 ]
stack commented on HBASE-1816:
------------------------------
More work on design: http://wiki.apache.org/hadoop/Hbase/MasterRewrite. At Patrick Hunt's prompting, will have a znode per table up in zk to hold schema and state (rather than one znode to hold all). Also, carried forward Patrick's suggestion for how to keep region state. It involves a flip on previous notion where online regions were in .META. somehow and zk just managed unassigned to open. Instead, .META. lists all regions. zk holds moving regions from unassigned -- i.e. mentioned in .META. but not in zk -- through to online (online is a state in zk).
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf, master-v2.patch, master-v3.patch, master-v4.patch, master.patch, meta-cleanup.patch
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1816) Master rewrite
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1816:
-------------------------
Attachment: master-v2.patch
More on the master cleanup theme. Not done yet.
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf, master-v2.patch, master.patch, meta-cleanup.patch
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1816) Master rewrite
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-1816:
----------------------------
Assignee: stack
> Master rewrite
> --------------
>
> Key: HBASE-1816
> URL: https://issues.apache.org/jira/browse/HBASE-1816
> Project: Hadoop HBase
> Issue Type: Task
> Components: master
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: HBase Hackathon Notes - Sunday.pdf
>
>
> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.