You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Nitay Joffe (JIRA)" <ji...@apache.org> on 2009/02/19 01:51:02 UTC
[jira] Created: (HBASE-1205) RegionServers should find new master
when a new master comes up.
RegionServers should find new master when a new master comes up.
----------------------------------------------------------------
Key: HBASE-1205
URL: https://issues.apache.org/jira/browse/HBASE-1205
Project: Hadoop HBase
Issue Type: Improvement
Reporter: Nitay Joffe
Assignee: Nitay Joffe
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1205) RegionServers should find new master
when a new master comes up.
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12693954#action_12693954 ]
Andrew Purtell commented on HBASE-1205:
---------------------------------------
+1 on this patch.
+1 on opening an issue to let regionservers continue carrying region assignments from an earlier master.
> RegionServers should find new master when a new master comes up.
> ----------------------------------------------------------------
>
> Key: HBASE-1205
> URL: https://issues.apache.org/jira/browse/HBASE-1205
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: hbase-1205.patch, hbase-nitay-regionserver-aa0-000-12.u.powerset.com.log
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-1205) RegionServers should find
new master when a new master comes up.
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695206#action_12695206 ]
Andrew Purtell edited comment on HBASE-1205 at 4/2/09 6:25 PM:
---------------------------------------------------------------
Committed to trunk. Passes all local tests.
Included stack's suggestion about info level logging on master change.
was (Author: apurtell):
Committed to trunk. Passes all local tests.
> RegionServers should find new master when a new master comes up.
> ----------------------------------------------------------------
>
> Key: HBASE-1205
> URL: https://issues.apache.org/jira/browse/HBASE-1205
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: hbase-1205.patch, hbase-nitay-regionserver-aa0-000-12.u.powerset.com.log
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1205) RegionServers should find new master
when a new master comes up.
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695206#action_12695206 ]
Andrew Purtell commented on HBASE-1205:
---------------------------------------
Committed to trunk. Passes all local tests.
> RegionServers should find new master when a new master comes up.
> ----------------------------------------------------------------
>
> Key: HBASE-1205
> URL: https://issues.apache.org/jira/browse/HBASE-1205
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: hbase-1205.patch, hbase-nitay-regionserver-aa0-000-12.u.powerset.com.log
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1205) RegionServers should find new master
when a new master comes up.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12693902#action_12693902 ]
stack commented on HBASE-1205:
------------------------------
Minor (Don't bother making another patch -- I can do it when I commit):
Switching masters is probably deserving of info-level logging.
Is that all it takes to make the regionservers notice a new master and then sign on? What happens to the regions the regionserver was carrying? Do they carry on serving them? From attached log, it looks like we start over?
> RegionServers should find new master when a new master comes up.
> ----------------------------------------------------------------
>
> Key: HBASE-1205
> URL: https://issues.apache.org/jira/browse/HBASE-1205
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: hbase-1205.patch, hbase-nitay-regionserver-aa0-000-12.u.powerset.com.log
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1205) RegionServers should find new master
when a new master comes up.
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nitay Joffe updated HBASE-1205:
-------------------------------
Fix Version/s: 0.20.0
Affects Version/s: 0.20.0
Status: Patch Available (was: Open)
> RegionServers should find new master when a new master comes up.
> ----------------------------------------------------------------
>
> Key: HBASE-1205
> URL: https://issues.apache.org/jira/browse/HBASE-1205
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: hbase-1205.patch, hbase-nitay-regionserver-aa0-000-12.u.powerset.com.log
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1205) RegionServers should find new master
when a new master comes up.
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1205:
----------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
> RegionServers should find new master when a new master comes up.
> ----------------------------------------------------------------
>
> Key: HBASE-1205
> URL: https://issues.apache.org/jira/browse/HBASE-1205
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: hbase-1205.patch, hbase-nitay-regionserver-aa0-000-12.u.powerset.com.log
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1205) RegionServers should find new master
when a new master comes up.
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nitay Joffe updated HBASE-1205:
-------------------------------
Attachment: hbase-nitay-regionserver-aa0-000-12.u.powerset.com.log
hbase-1205.patch
In this patch:
We register the HRegionServer as a watcher on the master address ephemeral ZNode in ZooKeeper. The HRegionServer watches for "NodeCreated" events on that ZNode, which means a new master has come up and written its address to the ZNode. When that event is received, we call getMaster() to set the hbaseMaster instance variable to the new master address. Any current operations going on which were using the old master will fail their retries eventually and get automatically switched over to the new master next time.
I made the sleeper variable that's used in run() an instance variable so that I can use it in other places like watchMasterAddress().
I tested this by taking down the master on my 4 node cluster and bringing it up elsewhere. I've attached the log from a regionserver. This is the cool part:
{code}
2009-03-30 07:57:13,977 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 0 time(s).
2009-03-30 07:57:14,987 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 1 time(s).
2009-03-30 07:57:15,232 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: SyncConnected, type: NodeCreated, path: /hbase/master
2009-03-30 07:57:15,233 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/master got 208.76.44.139:60007
2009-03-30 07:57:15,233 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at 208.76.44.139:60007 that we are up
{code}
> RegionServers should find new master when a new master comes up.
> ----------------------------------------------------------------
>
> Key: HBASE-1205
> URL: https://issues.apache.org/jira/browse/HBASE-1205
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Attachments: hbase-1205.patch, hbase-nitay-regionserver-aa0-000-12.u.powerset.com.log
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1205) RegionServers should find new master
when a new master comes up.
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12693908#action_12693908 ]
Nitay Joffe commented on HBASE-1205:
------------------------------------
Yes, this doesn't change what the master does on startup. It just makes the regionserver find the new master so it doesn't go retrying on the dead master endlessly. That means we'll still go reassigning all of the regions, which is not ideal, as you mention. Shall I open a separate issue for it?
> RegionServers should find new master when a new master comes up.
> ----------------------------------------------------------------
>
> Key: HBASE-1205
> URL: https://issues.apache.org/jira/browse/HBASE-1205
> Project: Hadoop HBase
> Issue Type: Improvement
> Affects Versions: 0.20.0
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
> Fix For: 0.20.0
>
> Attachments: hbase-1205.patch, hbase-nitay-regionserver-aa0-000-12.u.powerset.com.log
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1205) RegionServers should find new master
when a new master comes up.
Posted by "Nitay Joffe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675088#action_12675088 ]
Nitay Joffe commented on HBASE-1205:
------------------------------------
Either that, or each region server can put a watcher on the master address node so that when it is deleted/recreated they update their stored master address.
> RegionServers should find new master when a new master comes up.
> ----------------------------------------------------------------
>
> Key: HBASE-1205
> URL: https://issues.apache.org/jira/browse/HBASE-1205
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1205) RegionServers should find new master
when a new master comes up.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675087#action_12675087 ]
stack commented on HBASE-1205:
------------------------------
Is this about regionserver polling zk when without a master?
> RegionServers should find new master when a new master comes up.
> ----------------------------------------------------------------
>
> Key: HBASE-1205
> URL: https://issues.apache.org/jira/browse/HBASE-1205
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Nitay Joffe
> Assignee: Nitay Joffe
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.