You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Andrew Purtell <ap...@apache.org> on 2009/11/18 16:43:39 UTC
Re: brief note on 1015 (was Re: [jira] Commented: (HBASE-1816) Master rewrite)
Glad to see the change to keeping online regions in zk also. See the tail of HBASE-1015. C/C++ client does not want to talk to the master now that we have zk.
On a related note I looked at avro trunk and its C language binding is NOT ready. We may have no better option than thrift depending on how long this should wait.
- Andy
On Tue Nov 17th, 2009 3:44 PM PST stack (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779207#action_12779207 ]
>
>stack commented on HBASE-1816:
>------------------------------
>
>More work on design: http://wiki.apache.org/hadoop/Hbase/MasterRewrite. At Patrick Hunt's prompting, will have a znode per table up in zk to hold schema and state (rather than one znode to hold all). Also, carried forward Patrick's suggestion for how to keep region state. It involves a flip on previous notion where online regions were in .META. somehow and zk just managed unassigned to open. Instead, .META. lists all regions. zk holds moving regions from unassigned -- i.e. mentioned in .META. but not in zk -- through to online (online is a state in zk).
>
>> Master rewrite
>> --------------
>>
>> Key: HBASE-1816
>> URL: https://issues.apache.org/jira/browse/HBASE-1816
>> Project: Hadoop HBase
>> Issue Type: Task
>> Components: master
>> Reporter: stack
>> Assignee: stack
>> Priority: Blocker
>> Fix For: 0.21.0
>>
>> Attachments: HBase Hackathon Notes - Sunday.pdf, master-v2.patch, master-v3.patch, master-v4.patch, master.patch, meta-cleanup.patch
>>
>>
>> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
>
>--
>This message is automatically generated by JIRA.
>-
>You can reply to this email to add a comment to the issue online.
>
Re: brief note on 1015 (was Re: [jira] Commented: (HBASE-1816) Master
rewrite)
Posted by Ryan Rawson <ry...@gmail.com>.
Just to be clear, we are currently master independent for reads/writes
right now. The ROOT table location is in ZK, then we only talk to
regionservers from then on in.
Doing all region location finding in ZK isn't currently possible
because there is no 'search' type function, you'd have to retrieve all
regions for a table and do the search locally. There was talk about
putting in this capability into ZK, but I don't know the current
status of that.
On Wed, Nov 18, 2009 at 11:16 AM, Andrew Purtell <ap...@apache.org> wrote:
>> These things are orthogonal [...] It isn't there to facilitate region finding - META will still be doing that for a while.
>
> There's no reason region finding can be done with ZK given Stack's description of what he's doing. So, 1015 should use it to avoid needing to talk to a master. There are multiple benefits there.
>
> I agree about avro benefits. If users don't have any compelling reason to want 1015 in the 0.21 timeframe, I'm happy to push it out.
>
> - Andy
>
>
>
>
>
> ________________________________
> From: Ryan Rawson <ry...@gmail.com>
> To: hbase-dev@hadoop.apache.org
> Sent: Wed, November 18, 2009 9:53:31 AM
> Subject: Re: brief note on 1015 (was Re: [jira] Commented: (HBASE-1816) Master rewrite)
>
> These things are orthogonal, the regions going into ZK is to get rid
> of the 'regions in transition' internal datastructure of the master.
> It isn't there to facilitate region finding - META will still be doing
> that for a while.
>
> As for Avro, there are other benefits other than a C/C++ language API,
> so assuming that the Java bindings seems ready come 0.22 time, I would
> propose we do that, even if the C version isn't ready. At which point
> the C version becomes ready, then hbase will be there for it.
>
> But all of this stuff is months away. There is an 'avro hackathon'
> tomorrow night. Some people are interested in a PHP version somehow
> (using C possibly).
>
> -ryan
>
> On Wed, Nov 18, 2009 at 7:43 AM, Andrew Purtell <ap...@apache.org> wrote:
>> Glad to see the change to keeping online regions in zk also. See the tail of HBASE-1015. C/C++ client does not want to talk to the master now that we have zk.
>>
>> On a related note I looked at avro trunk and its C language binding is NOT ready. We may have no better option than thrift depending on how long this should wait.
>>
>> - Andy
>>
>>
>> On Tue Nov 17th, 2009 3:44 PM PST stack (JIRA) wrote:
>>
>>>
>>> [ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779207#action_12779207 ]
>>>
>>>stack commented on HBASE-1816:
>>>------------------------------
>>>
>>>More work on design: http://wiki.apache.org/hadoop/Hbase/MasterRewrite. At Patrick Hunt's prompting, will have a znode per table up in zk to hold schema and state (rather than one znode to hold all). Also, carried forward Patrick's suggestion for how to keep region state. It involves a flip on previous notion where online regions were in .META. somehow and zk just managed unassigned to open. Instead, .META. lists all regions. zk holds moving regions from unassigned -- i.e. mentioned in .META. but not in zk -- through to online (online is a state in zk).
>>>
>>>> Master rewrite
>>>> --------------
>>>>
>>>> Key: HBASE-1816
>>>> URL: https://issues.apache.org/jira/browse/HBASE-1816
>>>> Project: Hadoop HBase
>>>> Issue Type: Task
>>>> Components: master
>>>> Reporter: stack
>>>> Assignee: stack
>>>> Priority: Blocker
>>>> Fix For: 0.21.0
>>>>
>>>> Attachments: HBase Hackathon Notes - Sunday.pdf, master-v2.patch, master-v3.patch, master-v4.patch, master.patch, meta-cleanup.patch
>>>>
>>>>
>>>> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
>>>
>>>--
>>>This message is automatically generated by JIRA.
>>>-
>>>You can reply to this email to add a comment to the issue online.
>>>
>>
>>
>>
>>
>>
>
>
>
>
Re: brief note on 1015 (was Re: [jira] Commented: (HBASE-1816) Master rewrite)
Posted by Andrew Purtell <ap...@apache.org>.
> These things are orthogonal [...] It isn't there to facilitate region finding - META will still be doing that for a while.
There's no reason region finding can be done with ZK given Stack's description of what he's doing. So, 1015 should use it to avoid needing to talk to a master. There are multiple benefits there.
I agree about avro benefits. If users don't have any compelling reason to want 1015 in the 0.21 timeframe, I'm happy to push it out.
- Andy
________________________________
From: Ryan Rawson <ry...@gmail.com>
To: hbase-dev@hadoop.apache.org
Sent: Wed, November 18, 2009 9:53:31 AM
Subject: Re: brief note on 1015 (was Re: [jira] Commented: (HBASE-1816) Master rewrite)
These things are orthogonal, the regions going into ZK is to get rid
of the 'regions in transition' internal datastructure of the master.
It isn't there to facilitate region finding - META will still be doing
that for a while.
As for Avro, there are other benefits other than a C/C++ language API,
so assuming that the Java bindings seems ready come 0.22 time, I would
propose we do that, even if the C version isn't ready. At which point
the C version becomes ready, then hbase will be there for it.
But all of this stuff is months away. There is an 'avro hackathon'
tomorrow night. Some people are interested in a PHP version somehow
(using C possibly).
-ryan
On Wed, Nov 18, 2009 at 7:43 AM, Andrew Purtell <ap...@apache.org> wrote:
> Glad to see the change to keeping online regions in zk also. See the tail of HBASE-1015. C/C++ client does not want to talk to the master now that we have zk.
>
> On a related note I looked at avro trunk and its C language binding is NOT ready. We may have no better option than thrift depending on how long this should wait.
>
> - Andy
>
>
> On Tue Nov 17th, 2009 3:44 PM PST stack (JIRA) wrote:
>
>>
>> [ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779207#action_12779207 ]
>>
>>stack commented on HBASE-1816:
>>------------------------------
>>
>>More work on design: http://wiki.apache.org/hadoop/Hbase/MasterRewrite. At Patrick Hunt's prompting, will have a znode per table up in zk to hold schema and state (rather than one znode to hold all). Also, carried forward Patrick's suggestion for how to keep region state. It involves a flip on previous notion where online regions were in .META. somehow and zk just managed unassigned to open. Instead, .META. lists all regions. zk holds moving regions from unassigned -- i.e. mentioned in .META. but not in zk -- through to online (online is a state in zk).
>>
>>> Master rewrite
>>> --------------
>>>
>>> Key: HBASE-1816
>>> URL: https://issues.apache.org/jira/browse/HBASE-1816
>>> Project: Hadoop HBase
>>> Issue Type: Task
>>> Components: master
>>> Reporter: stack
>>> Assignee: stack
>>> Priority: Blocker
>>> Fix For: 0.21.0
>>>
>>> Attachments: HBase Hackathon Notes - Sunday.pdf, master-v2.patch, master-v3.patch, master-v4.patch, master.patch, meta-cleanup.patch
>>>
>>>
>>> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
>>
>>--
>>This message is automatically generated by JIRA.
>>-
>>You can reply to this email to add a comment to the issue online.
>>
>
>
>
>
>
Re: brief note on 1015 (was Re: [jira] Commented: (HBASE-1816) Master
rewrite)
Posted by Ryan Rawson <ry...@gmail.com>.
These things are orthogonal, the regions going into ZK is to get rid
of the 'regions in transition' internal datastructure of the master.
It isn't there to facilitate region finding - META will still be doing
that for a while.
As for Avro, there are other benefits other than a C/C++ language API,
so assuming that the Java bindings seems ready come 0.22 time, I would
propose we do that, even if the C version isn't ready. At which point
the C version becomes ready, then hbase will be there for it.
But all of this stuff is months away. There is an 'avro hackathon'
tomorrow night. Some people are interested in a PHP version somehow
(using C possibly).
-ryan
On Wed, Nov 18, 2009 at 7:43 AM, Andrew Purtell <ap...@apache.org> wrote:
> Glad to see the change to keeping online regions in zk also. See the tail of HBASE-1015. C/C++ client does not want to talk to the master now that we have zk.
>
> On a related note I looked at avro trunk and its C language binding is NOT ready. We may have no better option than thrift depending on how long this should wait.
>
> - Andy
>
>
> On Tue Nov 17th, 2009 3:44 PM PST stack (JIRA) wrote:
>
>>
>> [ https://issues.apache.org/jira/browse/HBASE-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779207#action_12779207 ]
>>
>>stack commented on HBASE-1816:
>>------------------------------
>>
>>More work on design: http://wiki.apache.org/hadoop/Hbase/MasterRewrite. At Patrick Hunt's prompting, will have a znode per table up in zk to hold schema and state (rather than one znode to hold all). Also, carried forward Patrick's suggestion for how to keep region state. It involves a flip on previous notion where online regions were in .META. somehow and zk just managed unassigned to open. Instead, .META. lists all regions. zk holds moving regions from unassigned -- i.e. mentioned in .META. but not in zk -- through to online (online is a state in zk).
>>
>>> Master rewrite
>>> --------------
>>>
>>> Key: HBASE-1816
>>> URL: https://issues.apache.org/jira/browse/HBASE-1816
>>> Project: Hadoop HBase
>>> Issue Type: Task
>>> Components: master
>>> Reporter: stack
>>> Assignee: stack
>>> Priority: Blocker
>>> Fix For: 0.21.0
>>>
>>> Attachments: HBase Hackathon Notes - Sunday.pdf, master-v2.patch, master-v3.patch, master-v4.patch, master.patch, meta-cleanup.patch
>>>
>>>
>>> An umbrella issue for master rewrite. Master rewrite is a new feature for 0.21 hbase. I'm attaching "Master Rewrite" notes taken at the SU SF hbase meetup made by JGray.
>>
>>--
>>This message is automatically generated by JIRA.
>>-
>>You can reply to this email to add a comment to the issue online.
>>
>
>
>
>
>