You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "mingjian (Created) (JIRA)" <ji...@apache.org> on 2011/11/09 04:25:51 UTC
[jira] [Created] (HBASE-4762) ROOT and META region never be
assigned if IOE throws in verifyRootRegionLocation
ROOT and META region never be assigned if IOE throws in verifyRootRegionLocation
--------------------------------------------------------------------------------
Key: HBASE-4762
URL: https://issues.apache.org/jira/browse/HBASE-4762
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: mingjian
Assignee: mingjian
Fix For: 0.94.0, 0.90.5
Patch in HBASE-3914 fixed root assigned in two regionservers. But it seemed like root region will never be assigned if verifyRootRegionLocation throws IOE.
Like following master logs:
{noformat}
2011-10-19 19:13:34,873 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_META_SERVER_S
HUTDOWN
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running yet
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1090)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:256)
at $Proxy7.getRegionInfo(Unknown Source)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRootRegionLocation(CatalogTracker.java:471)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.verifyAndAssignRoot(ServerShutdownHandler.java:90)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:126)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
After this, -ROOT-'s region won't be assigned, like this:
{noformat}
2011-10-19 19:18:40,000 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parent
Table=-ROOT-, metaLocation=address: dw79.kgb.sqa.cm4:60020, regioninfo: -ROOT-,,0.70236052, attempt=0 of 10 failed; retrying after s
leep of 1000 because: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2771)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1802)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:569)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1091)
{noformat}
So we should rewrite the verifyRootRegionLocation method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4762) ROOT and META region never be
assigned if IOE throws in verifyRootRegionLocation
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203811#comment-13203811 ]
stack commented on HBASE-4762:
------------------------------
This should be less likely in TRUNK, right RAM?, since we retry in TRUNK but not in 0.90.x
> ROOT and META region never be assigned if IOE throws in verifyRootRegionLocation
> --------------------------------------------------------------------------------
>
> Key: HBASE-4762
> URL: https://issues.apache.org/jira/browse/HBASE-4762
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.4
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.7
>
>
> Patch in HBASE-3914 fixed root assigned in two regionservers. But it seemed like root region will never be assigned if verifyRootRegionLocation throws IOE.
> Like following master logs:
> {noformat}
> 2011-10-19 19:13:34,873 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_META_SERVER_S
> HUTDOWN
> org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running yet
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1090)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:256)
> at $Proxy7.getRegionInfo(Unknown Source)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRootRegionLocation(CatalogTracker.java:471)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.verifyAndAssignRoot(ServerShutdownHandler.java:90)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:126)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> After this, -ROOT-'s region won't be assigned, like this:
> {noformat}
> 2011-10-19 19:18:40,000 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parent
> Table=-ROOT-, metaLocation=address: dw79.kgb.sqa.cm4:60020, regioninfo: -ROOT-,,0.70236052, attempt=0 of 10 failed; retrying after s
> leep of 1000 because: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2771)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1802)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:569)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1091)
> {noformat}
> So we should rewrite the verifyRootRegionLocation method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4762) ROOT and META region never be
assigned if IOE throws in verifyRootRegionLocation
Posted by "ramkrishna.s.vasudevan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-4762:
------------------------------------------
Fix Version/s: (was: 0.90.6)
0.90.7
Hadoop Flags: Reviewed
Moving to 0.90.7
> ROOT and META region never be assigned if IOE throws in verifyRootRegionLocation
> --------------------------------------------------------------------------------
>
> Key: HBASE-4762
> URL: https://issues.apache.org/jira/browse/HBASE-4762
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.4
> Reporter: mingjian
> Assignee: mingjian
> Fix For: 0.90.7
>
>
> Patch in HBASE-3914 fixed root assigned in two regionservers. But it seemed like root region will never be assigned if verifyRootRegionLocation throws IOE.
> Like following master logs:
> {noformat}
> 2011-10-19 19:13:34,873 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_META_SERVER_S
> HUTDOWN
> org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running yet
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1090)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:256)
> at $Proxy7.getRegionInfo(Unknown Source)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRegionLocation(CatalogTracker.java:424)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyRootRegionLocation(CatalogTracker.java:471)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.verifyAndAssignRoot(ServerShutdownHandler.java:90)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:126)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> After this, -ROOT-'s region won't be assigned, like this:
> {noformat}
> 2011-10-19 19:18:40,000 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: locateRegionInMeta parent
> Table=-ROOT-, metaLocation=address: dw79.kgb.sqa.cm4:60020, regioninfo: -ROOT-,,0.70236052, attempt=0 of 10 failed; retrying after s
> leep of 1000 because: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2771)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1802)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:569)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1091)
> {noformat}
> So we should rewrite the verifyRootRegionLocation method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira