You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2010/10/21 09:22:14 UTC
[jira] Created: (HBASE-3138) When new master joins running cluster
but meta is yanked from it as processing RIT, gets unexpected state
When new master joins running cluster but meta is yanked from it as processing RIT, gets unexpected state
---------------------------------------------------------------------------------------------------------
Key: HBASE-3138
URL: https://issues.apache.org/jira/browse/HBASE-3138
Project: HBase
Issue Type: Bug
Reporter: stack
Fix For: 0.90.0
Testing rolling restart i turned up the following condition.
Master is joining an extant cluster and is trying to clean up RIT. Then the server hosting .META. is shutdown in the middle of it all. Deal. Here is exception.
{code}
2010-10-21 06:45:58,592 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=sv2borg187,60020,1287643131919, region=efcd899283e96f20faa317772f52adca
2010-10-21 06:45:58,616 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Server not running
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2198)
at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1499)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:749)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
at $Proxy1.get(Unknown Source)
at org.apache.hadoop.hbase.catalog.MetaReader.getRegion(MetaReader.java:286)
at org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:250)
at org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:209)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:392)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:268)
2010-10-21 06:45:58,617 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3138) When new master joins running cluster
but meta is yanked from it as processing RIT, gets unexpected state
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3138:
-------------------------
Priority: Blocker (was: Major)
> When new master joins running cluster but meta is yanked from it as processing RIT, gets unexpected state
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3138
> URL: https://issues.apache.org/jira/browse/HBASE-3138
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.90.0
>
>
> Testing rolling restart i turned up the following condition.
> Master is joining an extant cluster and is trying to clean up RIT. Then the server hosting .META. is shutdown in the middle of it all. Deal. Here is exception.
> {code}
> 2010-10-21 06:45:58,592 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=sv2borg187,60020,1287643131919, region=efcd899283e96f20faa317772f52adca
> 2010-10-21 06:45:58,616 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Server not running
> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2198)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1499)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:749)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
> at $Proxy1.get(Unknown Source)
> at org.apache.hadoop.hbase.catalog.MetaReader.getRegion(MetaReader.java:286)
> at org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:250)
> at org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:209)
> at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:392)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:268)
> 2010-10-21 06:45:58,617 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3138) When new master joins running
cluster but meta is yanked from it as processing RIT, gets unexpected state
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929311#action_12929311 ]
stack commented on HBASE-3138:
------------------------------
Regionname in RIT is close to useless. Serialized HRI would for sure be better.
> When new master joins running cluster but meta is yanked from it as processing RIT, gets unexpected state
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3138
> URL: https://issues.apache.org/jira/browse/HBASE-3138
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.90.0
>
>
> Testing rolling restart i turned up the following condition.
> Master is joining an extant cluster and is trying to clean up RIT. Then the server hosting .META. is shutdown in the middle of it all. Deal. Here is exception.
> {code}
> 2010-10-21 06:45:58,592 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=sv2borg187,60020,1287643131919, region=efcd899283e96f20faa317772f52adca
> 2010-10-21 06:45:58,616 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Server not running
> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2198)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1499)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:749)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
> at $Proxy1.get(Unknown Source)
> at org.apache.hadoop.hbase.catalog.MetaReader.getRegion(MetaReader.java:286)
> at org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:250)
> at org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:209)
> at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:392)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:268)
> 2010-10-21 06:45:58,617 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3138) When new master joins running
cluster but meta is yanked from it as processing RIT, gets unexpected state
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927258#action_12927258 ]
Jonathan Gray commented on HBASE-3138:
--------------------------------------
This is a little tricky. What should we do when we get an exception processing a RIT during failover? We could just log it and move on. If we ensure that we put the state into RIT in-memory map as soon as possible, then even if we get an exception, we'll time it out later and we won't lose track.
> When new master joins running cluster but meta is yanked from it as processing RIT, gets unexpected state
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3138
> URL: https://issues.apache.org/jira/browse/HBASE-3138
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.90.0
>
>
> Testing rolling restart i turned up the following condition.
> Master is joining an extant cluster and is trying to clean up RIT. Then the server hosting .META. is shutdown in the middle of it all. Deal. Here is exception.
> {code}
> 2010-10-21 06:45:58,592 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=sv2borg187,60020,1287643131919, region=efcd899283e96f20faa317772f52adca
> 2010-10-21 06:45:58,616 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Server not running
> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2198)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1499)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:749)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
> at $Proxy1.get(Unknown Source)
> at org.apache.hadoop.hbase.catalog.MetaReader.getRegion(MetaReader.java:286)
> at org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:250)
> at org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:209)
> at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:392)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:268)
> 2010-10-21 06:45:58,617 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3138) When new master joins running
cluster but meta is yanked from it as processing RIT, gets unexpected state
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929304#action_12929304 ]
Jonathan Gray commented on HBASE-3138:
--------------------------------------
The issue is we don't have an HRI for the region-in-transition because we can't get to META. Without the HRI, we can't properly setup the state in RIT map.
We could serialize the HRI into the RIT node and use it there (right now we have byte [] regionName in there).
> When new master joins running cluster but meta is yanked from it as processing RIT, gets unexpected state
> ---------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3138
> URL: https://issues.apache.org/jira/browse/HBASE-3138
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.90.0
>
>
> Testing rolling restart i turned up the following condition.
> Master is joining an extant cluster and is trying to clean up RIT. Then the server hosting .META. is shutdown in the middle of it all. Deal. Here is exception.
> {code}
> 2010-10-21 06:45:58,592 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=sv2borg187,60020,1287643131919, region=efcd899283e96f20faa317772f52adca
> 2010-10-21 06:45:58,616 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Server not running
> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2198)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1499)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:749)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
> at $Proxy1.get(Unknown Source)
> at org.apache.hadoop.hbase.catalog.MetaReader.getRegion(MetaReader.java:286)
> at org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:250)
> at org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:209)
> at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:392)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:268)
> 2010-10-21 06:45:58,617 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.