You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jonathan Gray (JIRA)" <ji...@apache.org> on 2010/06/08 21:11:11 UTC

[jira] Created: (HBASE-2696) ZooKeeper cleanup and refactor

ZooKeeper cleanup and refactor
------------------------------

                 Key: HBASE-2696
                 URL: https://issues.apache.org/jira/browse/HBASE-2696
             Project: HBase
          Issue Type: Sub-task
          Components: zookeeper
            Reporter: Jonathan Gray
            Priority: Critical
             Fix For: 0.21.0


Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.

This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-2696:
-------------------------

          Status: Resolved  (was: Patch Available)
    Hadoop Flags: [Reviewed]
      Resolution: Fixed

Committed as part of HBASE-2692.  

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.90.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v3-NewClasses_RS.patch, HBASE-2696-part1-v4-NewClasses_RS_Tested.patch, HBASE-2696-part1-v5-NewClasses_RS_Tested.patch, HBASE-2696-part2-v2-FinishRS_FixClient.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-2696:
---------------------------------

    Attachment: HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch

Updated version.  Adds some stuff so we now abort when we get an unhandled KeeperException.

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888857#action_12888857 ] 

Jonathan Gray commented on HBASE-2696:
--------------------------------------

Next set of changes for ZK cleanup (following master startup cleanup in HBASE-2695).

This completes move from ZooKeeperWrapper to ZooKeeperWatcher (and all the new ZK bits) for the RS and Client.  There is still work to do in Master, that is next patch which hopefully comes today.

* Rename {{ServerStatus}} to {{ServerController}} (still have MasterStatus, that will be renamed in next patch)
** Client uses this now, maybe should be ProcessController?
* Adds a new client exception {{ZooKeeperConnectionException}} which is subclass of IOException.  Thrown from HTable and HBaseAdmin if no ZK connection
* Fixes client so that it can ride over master changes
** Uses {{MasterAddressManager}} already being used on RS
** {{HBaseAdmin}} uses accessor method to get master reference rather than a class variable
** Move all ZK interactions to go via ZKUtil/ZKWatcher not ZKWrapper
* Moves RS completely off of ZKWrapper and onto ZKUtil/ZKWatcher
* Adds lots of new methods to ZKUtil, every one with full javadoc

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.90.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v3-NewClasses_RS.patch, HBASE-2696-part1-v4-NewClasses_RS_Tested.patch, HBASE-2696-part1-v5-NewClasses_RS_Tested.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884965#action_12884965 ] 

Jonathan Gray commented on HBASE-2696:
--------------------------------------

There are many open issues still.

Basically all ZooKeeper operations will throw both {{KeeperException}} and {{InterruptedException}}.  Usually for each operation there are some expected KeeperExceptions like NODEEXISTS, BADVERSION, NONODE.  There are also some nastier ones like CONNECTIONLOSS.

So my thinking there is to explicitly deal with the KeeperExceptions that we expect, make sure to document the contract in the javadoc.  All other KeeperExceptions should abort the server.

Full list of KeeperException codes: http://hadoop.apache.org/zookeeper/docs/r3.3.1/api/org/apache/zookeeper/KeeperException.Code.html

As for InterruptedException, this is always one of those special things.  It will get clunky if we let them out of all the methods, can we get away with just returning null/false values and logging the exception? :)

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888861#action_12888861 ] 

HBase Review Board commented on HBASE-2696:
-------------------------------------------

Message from: stack@duboce.net

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/324/#review404
-----------------------------------------------------------

Ship it!


Looks good.  Nothing to test in this new code (though most is just replacing and adding new exception).


branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/ServerController.java
<http://review.hbase.org/r/324/#comment1716>

    Client "is a" ServerController or do you mean that you can do ServerController actions via client?  (I see later that whats going on here is that the client is proxying Server calls)
    
    
    Also, (don't kill me), but controller is bad name for this interface.  Going by the current set of methods in this interface, they are methods an actual controller would make use of; e.g. a "Controller" would decide its time to abort the server.
    
    I still think it should be 'Server' or 'Process'



branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/ServerController.java
<http://review.hbase.org/r/324/#comment1717>

    Is this javadoc right?  Mentions serverstatus



branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
<http://review.hbase.org/r/324/#comment1718>

    Should we remove this exception?  Or deprecate it?  The MasterNotRunning one?



branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
<http://review.hbase.org/r/324/#comment1719>

    Having this bit of code inside each method was really bad... thats good you fixed it



branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
<http://review.hbase.org/r/324/#comment1720>

    I'm for a super Interface.  Its odd have server-side in here.  How about Abortable Interface?



branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
<http://review.hbase.org/r/324/#comment1721>

    There is a Configurable interface up in hadoop (but also has setConf which you don't want I'd say)... just FYI.



branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/regionserver/MasterAddressManager.java
<http://review.hbase.org/r/324/#comment1722>

    Sorry, don't remember what I said in last review but did we agree that this class actually 'manages' the master address?  It has the watcher and updates itself if change?  That kinda thing?



branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/regionserver/RSZookeeperUpdater.java
<http://review.hbase.org/r/324/#comment1723>

    Call this RSZKUpdater to match your pattern naming classes that have to do w/ zk (ZKUtil, ZKConf).



branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
<http://review.hbase.org/r/324/#comment1726>

    Do you mean ensemble?



branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
<http://review.hbase.org/r/324/#comment1727>

    Should probably be ensemble but probably a big change doing that?


- stack





> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.90.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v3-NewClasses_RS.patch, HBASE-2696-part1-v4-NewClasses_RS_Tested.patch, HBASE-2696-part1-v5-NewClasses_RS_Tested.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-2696:
---------------------------------

    Attachment: HBASE-2696-part1-v3-NewClasses_RS.patch

Fixes TestMasterTransitions and maybe some other tests.

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v3-NewClasses_RS.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-2696:
---------------------------------

    Attachment:     (was: HBASE-2696-part1-v3-NewClasses_RS.patch)

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-2696:
---------------------------------

    Status: Patch Available  (was: Open)

Complete in master rewrite branch, keeping as patch available until merge with trunk.

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.90.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v3-NewClasses_RS.patch, HBASE-2696-part1-v4-NewClasses_RS_Tested.patch, HBASE-2696-part1-v5-NewClasses_RS_Tested.patch, HBASE-2696-part2-v2-FinishRS_FixClient.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888839#action_12888839 ] 

HBase Review Board commented on HBASE-2696:
-------------------------------------------

Message from: "Jonathan Gray" <jg...@apache.org>

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/324/
-----------------------------------------------------------

Review request for hbase, stack, Jean-Daniel Cryans, and Karthik Ranganathan.


Summary
-------

Next set of changes for ZK cleanup (following master startup cleanup in HBASE-2695).

This completes move from ZooKeeperWrapper to ZooKeeperWatcher (and all the new ZK bits) for the RS and Client.  There is still work to do in Master, that is next patch which hopefully comes today.

* Rename {{ServerStatus}} to {{ServerController}} (still have MasterStatus, that will be renamed in next patch)
** Client uses this now, maybe should be ProcessController?
* Adds a new client exception {{ZooKeeperConnectionException}} which is subclass of IOException.  Thrown from HTable and HBaseAdmin if no ZK connection
* Fixes client so that it can ride over master changes
** Uses {{MasterAddressManager}} already being used on RS
** {{HBaseAdmin}} uses accessor method to get master reference rather than a class variable
** Move all ZK interactions to go via ZKUtil/ZKWatcher not ZKWrapper
* Moves RS completely off of ZKWrapper and onto ZKUtil/ZKWatcher
* Adds lots of new methods to ZKUtil, every one with full javadoc


This addresses bug HBASE-2696.
    http://issues.apache.org/jira/browse/HBASE-2696


Diffs
-----

  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/ServerController.java PRE-CREATION 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/ServerStatus.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/ZooKeeperConnectionException.java PRE-CREATION 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/avro/AvroServer.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/client/HConnection.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/client/HTable.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/client/ServerConnectionManager.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/master/HMaster.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/master/MasterStatus.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/regionserver/MasterAddressManager.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/regionserver/RSZookeeperUpdater.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 964096 
  branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 964096 
  branches/0.90_master_rewrite/src/main/resources/hbase-webapps/master/zk.jsp 964096 
  branches/0.90_master_rewrite/src/main/resources/hbase-webapps/regionserver/regionserver.jsp 964096 
  branches/0.90_master_rewrite/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 964096 
  branches/0.90_master_rewrite/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java 964096 
  branches/0.90_master_rewrite/src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java 964096 
  branches/0.90_master_rewrite/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java 964096 
  branches/0.90_master_rewrite/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java 964096 

Diff: http://review.hbase.org/r/324/diff


Testing
-------

Had to fix up some of the ZK unit tests to work on new ZKUtil/ZKWatcher, all those passing.  Existing unit tests on MasterAddressManager.

Test suite seems to be passing, underway.


Thanks,

Jonathan




> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.90.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v3-NewClasses_RS.patch, HBASE-2696-part1-v4-NewClasses_RS_Tested.patch, HBASE-2696-part1-v5-NewClasses_RS_Tested.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888910#action_12888910 ] 

HBase Review Board commented on HBASE-2696:
-------------------------------------------

Message from: "Jonathan Gray" <jg...@apache.org>


bq.  On 2010-07-15 10:12:34, stack wrote:
bq.  > Looks good.  Nothing to test in this new code (though most is just replacing and adding new exception).

Yup.  It also fixes client to follow new masters and adds in a bunch of new methods to ZKUtil (unraveling ZKWrapper mess).

Next patch is finishing Master side and should be able to completely remove ZKWrapper and _all_ instances of Watchers/process() methods besides in ZooKeeperWatcher itself.

Thanks for review!


bq.  On 2010-07-15 10:12:34, stack wrote:
bq.  > branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/ServerController.java, line 27
bq.  > <http://review.hbase.org/r/324/diff/2/?file=2813#file2813line27>
bq.  >
bq.  >     Client "is a" ServerController or do you mean that you can do ServerController actions via client?  (I see later that whats going on here is that the client is proxying Server calls)
bq.  >     
bq.  >     
bq.  >     Also, (don't kill me), but controller is bad name for this interface.  Going by the current set of methods in this interface, they are methods an actual controller would make use of; e.g. a "Controller" would decide its time to abort the server.
bq.  >     
bq.  >     I still think it should be 'Server' or 'Process'

Makes sense and I see the confusion.

I think what you say below, an Abortable interface, probably makes good sense as the common base class for all of these.  Server anything is bad on client unless it does it _to_ the server, which is not the case here.  Will do this on the next patch.


bq.  On 2010-07-15 10:12:34, stack wrote:
bq.  > branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/ServerController.java, line 46
bq.  > <http://review.hbase.org/r/324/diff/2/?file=2813#file2813line46>
bq.  >
bq.  >     Is this javadoc right?  Mentions serverstatus

Will cleanup once we finalize


bq.  On 2010-07-15 10:12:34, stack wrote:
bq.  > branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java, line 325
bq.  > <http://review.hbase.org/r/324/diff/2/?file=2817#file2817line325>
bq.  >
bq.  >     Should we remove this exception?  Or deprecate it?  The MasterNotRunning one?

No, this is still valid.

Clients have two general exceptions now.  HTable will throw ZooKeeperConnectionException.  HBaseAdmin will throw ZK and MasterNotRunningException.

MNRE is for the case when ZK is up but either has no master address in it or is unable to connect to the master address in zk.


bq.  On 2010-07-15 10:12:34, stack wrote:
bq.  > branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 1681
bq.  > <http://review.hbase.org/r/324/diff/2/?file=2819#file2819line1681>
bq.  >
bq.  >     I'm for a super Interface.  Its odd have server-side in here.  How about Abortable Interface?

+1


bq.  On 2010-07-15 10:12:34, stack wrote:
bq.  > branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 1697
bq.  > <http://review.hbase.org/r/324/diff/2/?file=2819#file2819line1697>
bq.  >
bq.  >     There is a Configurable interface up in hadoop (but also has setConf which you don't want I'd say)... just FYI.

Once we add Abortable from above, this will get removed.


bq.  On 2010-07-15 10:12:34, stack wrote:
bq.  > branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/regionserver/MasterAddressManager.java, line 20
bq.  > <http://review.hbase.org/r/324/diff/2/?file=2825#file2825line20>
bq.  >
bq.  >     Sorry, don't remember what I said in last review but did we agree that this class actually 'manages' the master address?  It has the watcher and updates itself if change?  That kinda thing?

Yes, it actually manages it.  You point it at ZK and it exposes a method getMasterAddress() which will give you the up-to-date address from ZK (it doesn't actually read from ZK when you hit the method, it's being kept up to date automagically)


bq.  On 2010-07-15 10:12:34, stack wrote:
bq.  > branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/regionserver/RSZookeeperUpdater.java, line 1
bq.  > <http://review.hbase.org/r/324/diff/2/?file=2826#file2826line1>
bq.  >
bq.  >     Call this RSZKUpdater to match your pattern naming classes that have to do w/ zk (ZKUtil, ZKConf).

I'm hoping to rip this apart and put it into Handlers where it belongs.

Right now it's kinda mixed, some stuff is ZK some stuff is ZooKeeper.  ZooKeeper is just so friggin long ;)


bq.  On 2010-07-15 10:12:34, stack wrote:
bq.  > branches/0.90_master_rewrite/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java, line 54
bq.  > <http://review.hbase.org/r/324/diff/2/?file=2830#file2830line54>
bq.  >
bq.  >     Do you mean ensemble?

I used previous naming convention (all the existing ZK methods are about getting the quorum string).  You are probably right, it's the ensemble but would change a decent bit of existing code so not sure if we should change or not.


- Jonathan


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/324/#review404
-----------------------------------------------------------





> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.90.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v3-NewClasses_RS.patch, HBASE-2696-part1-v4-NewClasses_RS_Tested.patch, HBASE-2696-part1-v5-NewClasses_RS_Tested.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-2696:
---------------------------------

    Attachment: HBASE-2696-part1-NewClasses_NotIntegrated.patch

First patch of ZK cleanup.  I wanted to cut a patch and get this committed before I really rip stuff up.  This patch does not yet touch the existing ZooKeeperWrapper.

* Adds three new classes that will mostly replace current {{ZooKeeperWrapper}}:
** *{{ZooKeeperWatcher}}* acts as the single watcher for a process and also holds the ZooKeeper connection itself (instantiated class, no longer singleton like ZKWrapper).  All ZK connection events are handled in here, normal events are passed along to any registered {{ZooKeeperListeners}}.
** *{{ZooKeeperListener}}* is a base class that acts as hbase-internal ZK Watchers.  Consumers of ZK events will extend the class and override any of the events they are interested in.
** *{{ZKUtil}}* will contain static methods that actually do things like read/write znodes, list children, etc... These methods will take a {{ZooKeeperWatcher}}.
* Breaks out parts of {{HQuorumPeer}} into new {{ZKConfig}} class that just deals with the parsing of stuff


> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-2696:
---------------------------------

    Attachment: HBASE-2696-part1-v5-NewClasses_RS_Tested.patch

The patch actually applied to the branch this morning.

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v3-NewClasses_RS.patch, HBASE-2696-part1-v4-NewClasses_RS_Tested.patch, HBASE-2696-part1-v5-NewClasses_RS_Tested.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885278#action_12885278 ] 

Todd Lipcon commented on HBASE-2696:
------------------------------------

bq. As for InterruptedException, this is always one of those special things. It will get clunky if we let them out of all the methods, can we get away with just returning null/false values and logging the exception? 

If you let interruption pass through, you should do Thread.currentThread().interrupt() to maintain interruption status.

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v3-NewClasses_RS.patch, HBASE-2696-part1-v4-NewClasses_RS_Tested.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-2696:
---------------------------------

    Component/s: master
                 regionserver

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.21.0
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885806#action_12885806 ] 

Jonathan Gray commented on HBASE-2696:
--------------------------------------

I re-enabled TestZooKeeper.testRegionServerSessionExpired() in the branch.  The only test that is still disabled is the master session expiration but that's because i removed the auto restart on the master.  After a discussion with jd we might be able to safely allow that.  In any case, need to extend the hbase testing toolkit to support multiple masters so we can test failover.

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v3-NewClasses_RS.patch, HBASE-2696-part1-v4-NewClasses_RS_Tested.patch, HBASE-2696-part1-v5-NewClasses_RS_Tested.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-2696:
---------------------------------

    Attachment: HBASE-2696-part1-v3-NewClasses_RS.patch

part1-v3 adds integration on the RS side and removes the singleton stuff for ZKW.  Adds a new class on the regionserver side {{MasterAddressManager}} which encapsulates the watching/setting of the active master and is of the new {{ZooKeeperListener}} style.  There's some partial hack-up on the master side to get things working in this mixed world.  Also disables the regionserver and master abort tests in TestZooKeeper (as we've discussed, masters and regionservers don't restart themselves anymore).

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v3-NewClasses_RS.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-2696:
---------------------------------

    Attachment: HBASE-2696-part2-v2-FinishRS_FixClient.patch

Patch committed to master rewrite branch.

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.90.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v3-NewClasses_RS.patch, HBASE-2696-part1-v4-NewClasses_RS_Tested.patch, HBASE-2696-part1-v5-NewClasses_RS_Tested.patch, HBASE-2696-part2-v2-FinishRS_FixClient.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray updated HBASE-2696:
---------------------------------

    Attachment: HBASE-2696-part1-v4-NewClasses_RS_Tested.patch

Adds new test {{TestMasterAddressManager}}.  All unit tests passing.

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2696-part1-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v2-NewClasses_NotIntegrated.patch, HBASE-2696-part1-v3-NewClasses_RS.patch, HBASE-2696-part1-v4-NewClasses_RS_Tested.patch
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HBASE-2696) ZooKeeper cleanup and refactor

Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Gray reassigned HBASE-2696:
------------------------------------

    Assignee: Jonathan Gray

> ZooKeeper cleanup and refactor
> ------------------------------
>
>                 Key: HBASE-2696
>                 URL: https://issues.apache.org/jira/browse/HBASE-2696
>             Project: HBase
>          Issue Type: Sub-task
>          Components: master, regionserver, zookeeper
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Critical
>             Fix For: 0.21.0
>
>
> Currently almost everything we do with ZooKeeper is stuffed into a single class {{ZookeeperWrapper}}.
> This issue will deal with cleaning up our usage of ZK, adding some new abstractions to help with the master changes, splitting up watchers from utility methods, and nailing down the contracts of our ZK methods with respect to setting watchers, throwing exceptions, etc...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.