You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2009/10/05 04:11:56 UTC
[jira] Created: (HBASE-1886) move table and column family
attributes to ZK to make them live
move table and column family attributes to ZK to make them live
---------------------------------------------------------------
Key: HBASE-1886
URL: https://issues.apache.org/jira/browse/HBASE-1886
Project: Hadoop HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Fix For: 0.21.0
This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
The new hierarchy for attributes in ZK is like:
{noformat}
/hbase
/table
/<table-name>
/attr
/<table-attr-name>
/<family-name>
/<family-attr-name>
{noformat}
As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1886:
----------------------------------
Fix Version/s: (was: 0.20.2)
Un-target branch after -2.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767400#action_12767400 ]
Andrew Purtell commented on HBASE-1886:
---------------------------------------
@JG: Good suggestions, thanks. And thanks for looking at the patch. I agree with all of your points except branch inclusion.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.2, 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1886:
----------------------------------
Fix Version/s: 0.20.2
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.2, 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769498#action_12769498 ]
Andrew Purtell commented on HBASE-1886:
---------------------------------------
Initially this was something incremental because I was thinking about branch. I'm okay if you want to close this as duplicate of the master rewrite issue. After input from JD and JG this went in that direction anyhow.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell reassigned HBASE-1886:
-------------------------------------
Assignee: Andrew Purtell
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769487#action_12769487 ]
stack commented on HBASE-1886:
------------------------------
I took a quick look at the patch. I've been trying to do a bit of thinking on this topic myself as part of master rewrite design writeup: http://wiki.apache.org/hadoop/Hbase/MasterRewrite (as part of the rewrite master issue). I'd like to be a little more radical and move schema to zk too.
After chatting w/ J-D, I'm thinking we'd have a single tables znode up in zk. Its content is a JSON object that is a list of tables in the cluster. This JSON object would have two attributes: status and schema. The status would have attributes read-only, disabled, no-flush (these states seem to be mutually exclusive but I don't want to disallow setting more than attribute on a table). The schema would have the table's schema in JSON.
Every RS would have a watcher on this znode -- just one watcher rather than one-per-attribute as you'd need using your scheme? If tables changed, regionservers would react by checking every region against the content of the tables file. Could get a little involved if its carrying 1k regions, there are 1k tables, and one little attribute changed... but won't be bad either since just a spin through in-memory content.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767638#action_12767638 ]
Andrew Purtell commented on HBASE-1886:
---------------------------------------
bq. if attributes are node names, they cannot just be byte[] can they? ZK api uses String.
Keys are passed through Bytes.toStringBinary before becoming node names. But also the path separator needs to be escaped. If only printable characters are used this is basically a no-op. (Need to check if toStringBinary escapes more than != 32 <= 127)
bq. I'd like to expand this conversation to be about all the stuff we want to move into ZK, how we will design our usage of ZK, etc..
Sure. Regarding attributes in particular, how about this:
{noformat}
/hbase
/table
/attr({'json':'table attributes'})
/families/<family-name>
/attr({'json':'family attributes'})
{noformat}
with all attributes for a table or family wrapped up in a single JSON body. Assumes the rest will be dropped in place around it.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767395#action_12767395 ]
Andrew Purtell commented on HBASE-1886:
---------------------------------------
bq. it's a new feature that changes stuff directly where we are the weakest eg .META. scanner.
I have an internal customer who would like live attributes in 0.20 branch. I hope we can work something out. Not sure what you mean about changing the META scanner... I don't recall making any changes there.
bq. our patch relies on the SyncTableAttributes called every now and then to sync the Master internal state with what clients may have changed in ZK
Only on shutdown presently.
bq. Thanks for starting work on this!
Thanks. It's definitely a work in progress. I'm hoping to have some time to iterate on it this week and will address all of your comments. Thanks for looking at the patch J-D!
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.2, 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767398#action_12767398 ]
Jonathan Gray commented on HBASE-1886:
--------------------------------------
One other thing. We might bump families into their own node under the table. I expect we will have other things we'd like to publish into ZK on a per-table basis (and probably per-family), like load, status, whatever.
{noformat}
/hbase
/table
/<table-name>
/families/<family-name>
/load({'json':'load info'})
{noformat}
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.2, 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767641#action_12767641 ]
Jonathan Gray commented on HBASE-1886:
--------------------------------------
Binary issues doesn't really matter because now we're putting attributes into json. Only remaining issue would be with table names and family names, however these are already limited to printable characters which should be fine going from byte[] -> String -> byte[] (and we'll need to exclude slashes as well), right?
Attributes into their own node sounds good.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-1886.
--------------------------
Resolution: Duplicate
Resolve as duplicate of hbase-1816.... will carry on discussion in here over in the "master rewrite" issue.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767383#action_12767383 ]
Jean-Daniel Cryans commented on HBASE-1886:
-------------------------------------------
I think this shouldn't be in the 0.20 branch, it's a new feature that changes stuff directly where we are the weakest eg .META. scanner.
About the patch.
- Minor: you put the client methods in HTable instead of HBaseAdmin; the way I see it HTable is used for gets/puts/scans but not for administrative purposes.
- I think ZKW should be refactored, your work shows it well.
- If I understand it well, your patch relies on the SyncTableAttributes called every now and then to sync the Master internal state with what clients may have changed in ZK. Instead, would it be better to use watchers?
Thanks for starting work on this!
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.2, 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767436#action_12767436 ]
Jonathan Gray commented on HBASE-1886:
--------------------------------------
Also, if attributes are node names, they cannot just be byte[] can they? ZK api uses String.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.2, 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767437#action_12767437 ]
Jean-Daniel Cryans commented on HBASE-1886:
-------------------------------------------
+1 on John's post.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.2, 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762068#action_12762068 ]
Andrew Purtell commented on HBASE-1886:
---------------------------------------
The master copies table and family attributes up to ZK out of the table descriptor during the initial MetaScanner scan and from createTable. Attributes are written back to all descriptors in all regions of the table in META by the Master when shutdown is first initiated. deleteTable causes appropriate deletions.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Fix For: 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767640#action_12767640 ]
Andrew Purtell commented on HBASE-1886:
---------------------------------------
Sorry the formatting in the above is off. Table attr and families root should be at the same level.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1886:
----------------------------------
Attachment: live_attributes.patch
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Fix For: 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767411#action_12767411 ]
Andrew Purtell commented on HBASE-1886:
---------------------------------------
Ah, now I see the (minor) disconnect. SyncTableAttributes is only for writing modified attributes as new HTD in all appropriate regions in META on master shutdown. (Misnamed/confusing?) Otherwise what is in ZK is live and canonical across rolling restarts, master fail overs, region server restarts, etc. I haven't yet gone in and replaced all attribute related references to HTD and HCD in master and regionserver with calls into the new ZK methods. That was going to be next. Now first up is a second take incorporating all feedback.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.2, 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767393#action_12767393 ]
Jonathan Gray commented on HBASE-1886:
--------------------------------------
Agree with JD on all points. Don't think this should be part of 0.20 branch, methods belong in HBaseAdmin, and we should be using ZK watchers properly not artificially synchronizing.
As for layout in ZK... Do we want this very hierarchical layout with a high number of total nodes and where nodes are actually storing a good bit of the data (attribute names, family names, etc)...
Another thought is to use JSON to store this stuff in fewer total nodes. Dealing with lots of hierarchy can make it much more confusing to interact with this stuff. Rather than just watching for changed data on a node (for example, change in table configuration) you have to watch for a change in the children of a node. The changed children event doesn't actually tell you what has changed, you have to do some kind of DIFF to know what's changed. To monitor for changes to a single attribute of a single family, you'd have to have watchers for every single attribute value node. As long as we have _some_ hierarchy, so there isn't just one huge JSON string, then we could make usage/monitoring much easier.
Sorry if above is confusing... But ideally we'd want to be able to actually monitor for live changes to any of this stuff. The more hierarchy and nodes we have (and the more we have to monitor for changes in the children of a node VS changes in the data of a node), the more difficult it will be.
{noformat}
/hbase
/table
/<table-name>({'json':'table attributes'})
/<family-name>({'json':'family attributes'})
{noformat}
Not sure how big of a difference that actually makes, but I've built a few systems around ZK and have definitely found reducing hierarchy and keeping stuff together in JSON has kept things easier to manage (when you don't have a reason to read the dimensions individually, so no downside to aggregating into a json string).
Good stuff, Andrew.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.2, 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1886) move table and column family
attributes to ZK to make them live
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767432#action_12767432 ]
Jonathan Gray commented on HBASE-1886:
--------------------------------------
I have a few reasons for not wanting this part of branch, though I understand you needing it for internal use.
- This is not a standalone issue... the design of this should be part of the larger master redesign / increased usage of ZK (create/enable/disable tables, region assignment, load balancing, etc)
- Putting this into branch while also moving forward with more changes for trunk could mean a fork in ZK usage between the two
- And strictly speaking, this is a major change, not a major bug, and really doesn't belong in a minor revision
I'd like to expand this conversation to be about all the stuff we want to move into ZK, how we will design our usage of ZK, etc... rather than focusing on this single change.
On the other hand, we haven't had much work done in trunk around this stuff... this could be a good first step. I just don't want to tie our hands or limit our thinking because we want to put this into branch.
> move table and column family attributes to ZK to make them live
> ---------------------------------------------------------------
>
> Key: HBASE-1886
> URL: https://issues.apache.org/jira/browse/HBASE-1886
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Fix For: 0.20.2, 0.21.0
>
> Attachments: live_attributes.patch
>
>
> This is a first cut at moving table and column family attributes up to ZK where they can be modified any time without requiring a disable/enable table cycle. Adds methods to ZooKeeperWrapper for support server side. Adds methods to HTableInterface for client side access. Client side talks directly to ZK.
> The new hierarchy for attributes in ZK is like:
> {noformat}
> /hbase
> /table
> /<table-name>
> /attr
> /<table-attr-name>
> /<family-name>
> /<family-attr-name>
> {noformat}
> As before attribute names and values are byte[], but now that they are hosted in ZK, the path delimiter ('/') cannot be used.
> Attached patch compiles but has not been tested. I put it up for a concept review. Next step before this is functional is to deprecate the attribute related methods on HTD and HCD and chase through the warnings to find all users who should be updated to use the ZK wrapper or new HTable methods instead. And, of course, then add test cases that successfully complete.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.