You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jonathan Gray (JIRA)" <ji...@apache.org> on 2010/06/08 20:41:57 UTC
[jira] Created: (HBASE-2694) Move RS to Master region open/close
messaging into ZooKeeper
Move RS to Master region open/close messaging into ZooKeeper
------------------------------------------------------------
Key: HBASE-2694
URL: https://issues.apache.org/jira/browse/HBASE-2694
Project: HBase
Issue Type: Sub-task
Components: master, regionserver
Reporter: Jonathan Gray
Priority: Critical
Fix For: 0.21.0
As a first step towards HBASE-2485, this issue is about changing the message flow of opening and closing of regions without actually changing the implementation of what happens on both the Master and RegionServer sides. This way we can debug the messaging changes before the introduction of more significant changes to the master architecture and handling of regions in transition.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2694) Move RS to Master region open/close
messaging into ZooKeeper
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-2694:
---------------------------------
Attachment: HBASE-2694-final.patch
Final patch for commit. Includes changes from Todd's comments, added a few licenses, and also fixed mixed newlines on final patch file.
> Move RS to Master region open/close messaging into ZooKeeper
> ------------------------------------------------------------
>
> Key: HBASE-2694
> URL: https://issues.apache.org/jira/browse/HBASE-2694
> Project: HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Reporter: Jonathan Gray
> Priority: Critical
> Fix For: 0.21.0
>
> Attachments: HBASE-2694-final.patch, HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging-v2.patch, HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging.patch
>
>
> As a first step towards HBASE-2485, this issue is about changing the message flow of opening and closing of regions without actually changing the implementation of what happens on both the Master and RegionServer sides. This way we can debug the messaging changes before the introduction of more significant changes to the master architecture and handling of regions in transition.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2694) Move RS to Master region open/close
messaging into ZooKeeper
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878134#action_12878134 ]
HBase Review Board commented on HBASE-2694:
-------------------------------------------
Message from: "Karthik Ranganathan" <ka...@gmail.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/171/
-----------------------------------------------------------
Review request for hbase, Todd Lipcon and stack.
Summary
-------
Moved all RS -> Master communication to go through ZK so that we can enable master failover.
This addresses bug HBASE-2694.
http://issues.apache.org/jira/browse/HBASE-2694
Diffs
-----
trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 953804
trunk/src/main/java/org/apache/hadoop/hbase/executor/HBaseEventHandler.java PRE-CREATION
trunk/src/main/java/org/apache/hadoop/hbase/executor/HBaseExecutorService.java PRE-CREATION
trunk/src/main/java/org/apache/hadoop/hbase/executor/NamedThreadFactory.java PRE-CREATION
trunk/src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionEventData.java PRE-CREATION
trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java 953804
trunk/src/main/java/org/apache/hadoop/hbase/master/ProcessRegionClose.java 953804
trunk/src/main/java/org/apache/hadoop/hbase/master/ProcessRegionOpen.java 953804
trunk/src/main/java/org/apache/hadoop/hbase/master/ProcessRegionStatusChange.java 953804
trunk/src/main/java/org/apache/hadoop/hbase/master/RegionManager.java 953804
trunk/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 953804
trunk/src/main/java/org/apache/hadoop/hbase/master/ZKMasterAddressWatcher.java 953804
trunk/src/main/java/org/apache/hadoop/hbase/master/ZKUnassignedWatcher.java PRE-CREATION
trunk/src/main/java/org/apache/hadoop/hbase/master/handler/MasterCloseRegionHandler.java PRE-CREATION
trunk/src/main/java/org/apache/hadoop/hbase/master/handler/MasterOpenRegionHandler.java PRE-CREATION
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 953804
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/RSZookeeperUpdater.java PRE-CREATION
trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/HQuorumPeer.java 953804
trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java 953804
trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 953804
trunk/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java 953804
trunk/src/test/java/org/apache/hadoop/hbase/master/TestMaster.java 953804
trunk/src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java PRE-CREATION
trunk/src/test/java/org/apache/hadoop/hbase/master/TestZKBasedReopenRegion.java PRE-CREATION
Diff: http://review.hbase.org/r/171/diff
Testing
-------
Unit tested.
Thanks,
Karthik
> Move RS to Master region open/close messaging into ZooKeeper
> ------------------------------------------------------------
>
> Key: HBASE-2694
> URL: https://issues.apache.org/jira/browse/HBASE-2694
> Project: HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Reporter: Jonathan Gray
> Priority: Critical
> Fix For: 0.21.0
>
> Attachments: HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging-v2.patch, HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging.patch
>
>
> As a first step towards HBASE-2485, this issue is about changing the message flow of opening and closing of regions without actually changing the implementation of what happens on both the Master and RegionServer sides. This way we can debug the messaging changes before the introduction of more significant changes to the master architecture and handling of regions in transition.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2694) Move RS to Master region open/close
messaging into ZooKeeper
Posted by "Karthik Ranganathan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Ranganathan updated HBASE-2694:
---------------------------------------
Attachment: HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging-v2.patch
Second pass at the patch. Incorporates changes from in-person review with Todd and Stack. Unit tests pass.
> Move RS to Master region open/close messaging into ZooKeeper
> ------------------------------------------------------------
>
> Key: HBASE-2694
> URL: https://issues.apache.org/jira/browse/HBASE-2694
> Project: HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Reporter: Jonathan Gray
> Priority: Critical
> Fix For: 0.21.0
>
> Attachments: HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging-v2.patch, HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging.patch
>
>
> As a first step towards HBASE-2485, this issue is about changing the message flow of opening and closing of regions without actually changing the implementation of what happens on both the Master and RegionServer sides. This way we can debug the messaging changes before the introduction of more significant changes to the master architecture and handling of regions in transition.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2694) Move RS to Master region open/close
messaging into ZooKeeper
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878171#action_12878171 ]
HBase Review Board commented on HBASE-2694:
-------------------------------------------
Message from: "Jonathan Gray" <jg...@apache.org>
bq. On 2010-06-11 17:35:09, Todd Lipcon wrote:
bq. > Skimmed it reasonably quickly, looks good.
bq. > Major concern is how often we catch and just log exceptions. In most cases I think these are errors we can't recover from and it's best to throw RTE and shut down the whole server.
bq. >
bq. > Several places missing license headers.
Thanks for review Todd.
In ZK refactor, we will do in-depth look at where we should be throwing RTE vs logging/swallowing vs throwing.
When we do reimplementation of the open/close and all that, will look at the one you pointed out.
Applied the following changes per your comments:
- Switched to foreach in event handler
- Switch to NONE.value
- Added TODOs for using guava and reviewing IOE vs LOG
- Added license headers to ZKUnassignedWatcher, TestZKBasedReopenRegion, TestZKBasedCloseRegion
- Added logging to noop closing method
Will put final patch up onto JIRA and commit. Thanks!
bq. On 2010-06-11 17:35:09, Todd Lipcon wrote:
bq. > trunk/src/main/java/org/apache/hadoop/hbase/master/RegionManager.java, line 1000
bq. > <http://review.hbase.org/r/171/diff/1/?file=1267#file1267line1000>
bq. >
bq. > are these IOEs here and above really recoverable in any way? How could they happen?
They wouldn't happen unless the Writable implementations were broken. Not sure what we should be doing here besides logging? Better to log and ignore or throw RTE to bring attention to something being very broken?
Adding TODO to review what we should do here
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/171/#review202
-----------------------------------------------------------
> Move RS to Master region open/close messaging into ZooKeeper
> ------------------------------------------------------------
>
> Key: HBASE-2694
> URL: https://issues.apache.org/jira/browse/HBASE-2694
> Project: HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Reporter: Jonathan Gray
> Priority: Critical
> Fix For: 0.21.0
>
> Attachments: HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging-v2.patch, HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging.patch
>
>
> As a first step towards HBASE-2485, this issue is about changing the message flow of opening and closing of regions without actually changing the implementation of what happens on both the Master and RegionServer sides. This way we can debug the messaging changes before the introduction of more significant changes to the master architecture and handling of regions in transition.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2694) Move RS to Master region open/close
messaging into ZooKeeper
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876761#action_12876761 ]
Jonathan Gray commented on HBASE-2694:
--------------------------------------
This issue will include:
- Introduction of an {{UNASSIGNED}} znode/directory into ZK which will contain a znode for each region in transition. Data of the region znodes will contain the state of the region (closing, closed, opening, opened)
- Introduction of a custom executor service, {{HBaseExecutorService}}, which is a special type of named thread pool which will be used to process events
- Introduction of {{HBaseEventHandler}} interface which is the type executed by the HBaseExecutorService
- Two event handlers, {{MasterOpenRegionHandler}} and {{MasterCloseRegionHandler}}, to handle open/close events on the master side. For this jira, they will actually not contain new logic and will just call the existing open/close implementations.
The existing message flow for moving a region from one RS to another (close then open):
# M -> RS1: MSG_REGION_CLOSE (via piggybacking on the response to the RS heartbeat)
# RS1 -> M: MSG_REPORT_CLOSE (on heartbeat)
# M: Offline in META
# M: MetaScanner picks up unassigned region
# M -> RS2: MSG_REGION_OPEN (via piggybacking on the response to the RS heartbeat)
# RS2 -> M: MSG_REPORT_PROCESS_OPEN (sent on each heartbeat until region is opened)
# RS2 -> M: MSG_REPORT_OPEN (on heartbeat)
# M: Add opened region to meta
The new message flow:
# M -> RS1: MSG_REGION_CLOSE (via piggybacking on the response to the RS heartbeat)
# RS1 -> ZK: Create ZNode {{/UNASSIGNED/regionname(CLOSING)}}
# ZK -> M: Detects new region in transition, sets watcher on regionname znode
# RS1 -> ZK: Update ZNode {{/UNASSIGNED/regionname(CLOSED)}}
# ZK -> M: Detects region as closed, offline in META
# M: MetaScanner picks up unassigned region
# M -> RS2: MSG_REGION_OPEN (via piggybacking on the response to the RS heartbeat)
# RS2 -> ZK: Update ZNode {{/UNASSIGNED/regionname(OPENING)}}
# RS2 -> ZK: Update ZNode {{/UNASSIGNED/regionname(OPENED)}}
# ZK -> M: Detects region as opened, add opened region to meta
# M -> ZK: Delete ZNode {{/UNASSIGNED/regionname}}
> Move RS to Master region open/close messaging into ZooKeeper
> ------------------------------------------------------------
>
> Key: HBASE-2694
> URL: https://issues.apache.org/jira/browse/HBASE-2694
> Project: HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Reporter: Jonathan Gray
> Priority: Critical
> Fix For: 0.21.0
>
>
> As a first step towards HBASE-2485, this issue is about changing the message flow of opening and closing of regions without actually changing the implementation of what happens on both the Master and RegionServer sides. This way we can debug the messaging changes before the introduction of more significant changes to the master architecture and handling of regions in transition.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2694) Move RS to Master region open/close
messaging into ZooKeeper
Posted by "Karthik Ranganathan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878273#action_12878273 ]
Karthik Ranganathan commented on HBASE-2694:
--------------------------------------------
Woohoo, patch finally committed :) Thanks for the reviews guys, and thanks for the follow up and commit Jonathan - that was super quick!
> Move RS to Master region open/close messaging into ZooKeeper
> ------------------------------------------------------------
>
> Key: HBASE-2694
> URL: https://issues.apache.org/jira/browse/HBASE-2694
> Project: HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Reporter: Jonathan Gray
> Assignee: Karthik Ranganathan
> Priority: Critical
> Fix For: 0.21.0
>
> Attachments: HBASE-2694-final.patch, HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging-v2.patch, HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging.patch
>
>
> As a first step towards HBASE-2485, this issue is about changing the message flow of opening and closing of regions without actually changing the implementation of what happens on both the Master and RegionServer sides. This way we can debug the messaging changes before the introduction of more significant changes to the master architecture and handling of regions in transition.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2694) Move RS to Master region open/close
messaging into ZooKeeper
Posted by "Karthik Ranganathan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Ranganathan updated HBASE-2694:
---------------------------------------
Attachment: HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging.patch
Adding the patch for this... not verified that the merge with the latest code is correct from my private branch.
In my branch (staler version of code), the unit tests are passing and the cluster load and verify test is passing.
> Move RS to Master region open/close messaging into ZooKeeper
> ------------------------------------------------------------
>
> Key: HBASE-2694
> URL: https://issues.apache.org/jira/browse/HBASE-2694
> Project: HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Reporter: Jonathan Gray
> Priority: Critical
> Fix For: 0.21.0
>
> Attachments: HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging.patch
>
>
> As a first step towards HBASE-2485, this issue is about changing the message flow of opening and closing of regions without actually changing the implementation of what happens on both the Master and RegionServer sides. This way we can debug the messaging changes before the introduction of more significant changes to the master architecture and handling of regions in transition.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2694) Move RS to Master region open/close
messaging into ZooKeeper
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray resolved HBASE-2694.
----------------------------------
Hadoop Flags: [Reviewed]
Assignee: Karthik Ranganathan
Resolution: Fixed
Committed final patch. Awesome work Karthik.
Thanks to stack, todd, kannan, and everyone else for all the time spent reviewing.
1 down, 6 to go :)
> Move RS to Master region open/close messaging into ZooKeeper
> ------------------------------------------------------------
>
> Key: HBASE-2694
> URL: https://issues.apache.org/jira/browse/HBASE-2694
> Project: HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Reporter: Jonathan Gray
> Assignee: Karthik Ranganathan
> Priority: Critical
> Fix For: 0.21.0
>
> Attachments: HBASE-2694-final.patch, HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging-v2.patch, HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging.patch
>
>
> As a first step towards HBASE-2485, this issue is about changing the message flow of opening and closing of regions without actually changing the implementation of what happens on both the Master and RegionServer sides. This way we can debug the messaging changes before the introduction of more significant changes to the master architecture and handling of regions in transition.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2694) Move RS to Master region open/close
messaging into ZooKeeper
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878161#action_12878161 ]
HBase Review Board commented on HBASE-2694:
-------------------------------------------
Message from: "Todd Lipcon" <to...@cloudera.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.hbase.org/r/171/#review202
-----------------------------------------------------------
Ship it!
Skimmed it reasonably quickly, looks good.
Major concern is how often we catch and just log exceptions. In most cases I think these are errors we can't recover from and it's best to throw RTE and shut down the whole server.
Several places missing license headers.
trunk/src/main/java/org/apache/hadoop/hbase/executor/HBaseEventHandler.java
<http://review.hbase.org/r/171/#comment918>
use foreach form?
trunk/src/main/java/org/apache/hadoop/hbase/executor/HBaseEventHandler.java
<http://review.hbase.org/r/171/#comment919>
foreach form
trunk/src/main/java/org/apache/hadoop/hbase/executor/HBaseExecutorService.java
<http://review.hbase.org/r/171/#comment920>
== NONE.value, or if (this == NONE)?
trunk/src/main/java/org/apache/hadoop/hbase/executor/NamedThreadFactory.java
<http://review.hbase.org/r/171/#comment921>
You can just use this one:
http://guava-libraries.googlecode.com/svn/trunk/javadoc/com/google/common/util/concurrent/NamingThreadFactory.html
trunk/src/main/java/org/apache/hadoop/hbase/master/RegionManager.java
<http://review.hbase.org/r/171/#comment922>
are these IOEs here and above really recoverable in any way? How could they happen?
trunk/src/main/java/org/apache/hadoop/hbase/master/ZKUnassignedWatcher.java
<http://review.hbase.org/r/171/#comment923>
license
trunk/src/main/java/org/apache/hadoop/hbase/master/handler/MasterOpenRegionHandler.java
<http://review.hbase.org/r/171/#comment924>
maybe add a LOG.debug here at least for now
trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
<http://review.hbase.org/r/171/#comment925>
again, are these LOG.errors really recoverable things?
trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java
<http://review.hbase.org/r/171/#comment926>
recoverable?
trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWrapper.java
<http://review.hbase.org/r/171/#comment927>
can't wait til we refactor all this stuff
trunk/src/test/java/org/apache/hadoop/hbase/master/TestZKBasedCloseRegion.java
<http://review.hbase.org/r/171/#comment928>
license
- Todd
> Move RS to Master region open/close messaging into ZooKeeper
> ------------------------------------------------------------
>
> Key: HBASE-2694
> URL: https://issues.apache.org/jira/browse/HBASE-2694
> Project: HBase
> Issue Type: Sub-task
> Components: master, regionserver
> Reporter: Jonathan Gray
> Priority: Critical
> Fix For: 0.21.0
>
> Attachments: HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging-v2.patch, HBASE-2694-OPENSOURCE-TRUNK-zk-based-messaging.patch
>
>
> As a first step towards HBASE-2485, this issue is about changing the message flow of opening and closing of regions without actually changing the implementation of what happens on both the Master and RegionServer sides. This way we can debug the messaging changes before the introduction of more significant changes to the master architecture and handling of regions in transition.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.