You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jonathan Gray (JIRA)" <ji...@apache.org> on 2010/11/01 05:05:23 UTC
[jira] Created: (HBASE-3182) If server hosting META dies or is
stopping while processing another server shutdown, IOE accessing META stop
shutdown handler from finishing
If server hosting META dies or is stopping while processing another server shutdown, IOE accessing META stop shutdown handler from finishing
--------------------------------------------------------------------------------------------------------------------------------------------
Key: HBASE-3182
URL: https://issues.apache.org/jira/browse/HBASE-3182
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.0
Reporter: Jonathan Gray
Priority: Critical
Fix For: 0.90.0
In TestRollingRestart, there is a test which kills server hosting ROOT then immediately kills server hosting META. In a recent run this turned up a small race condition if the server hosting META is closing while we process shutdown of server hosting ROOT.
{noformat}
2010-10-31 20:41:34,621 ERROR [MASTER_META_SERVER_OPERATIONS-dev692.sf2p.facebook.com:54989-0] executor.EventHandler(154): Caught throwable while processing event M_META_SERVER_SHUTDOWN
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Server not running
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2216)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1652)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:749)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
at $Proxy8.openScanner(Unknown Source)
at org.apache.hadoop.hbase.catalog.MetaReader.getServerUserRegions(MetaReader.java:495)
at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:125)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-3182) If server hosting META dies or is
stopping while processing another server shutdown, IOE accessing META stop
shutdown handler from finishing
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray reassigned HBASE-3182:
------------------------------------
Assignee: Jonathan Gray
> If server hosting META dies or is stopping while processing another server shutdown, IOE accessing META stop shutdown handler from finishing
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3182
> URL: https://issues.apache.org/jira/browse/HBASE-3182
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Critical
> Fix For: 0.90.0
>
>
> In TestRollingRestart, there is a test which kills server hosting ROOT then immediately kills server hosting META. In a recent run this turned up a small race condition if the server hosting META is closing while we process shutdown of server hosting ROOT.
> {noformat}
> 2010-10-31 20:41:34,621 ERROR [MASTER_META_SERVER_OPERATIONS-dev692.sf2p.facebook.com:54989-0] executor.EventHandler(154): Caught throwable while processing event M_META_SERVER_SHUTDOWN
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Server not running
> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2216)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1652)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:749)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
> at $Proxy8.openScanner(Unknown Source)
> at org.apache.hadoop.hbase.catalog.MetaReader.getServerUserRegions(MetaReader.java:495)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:125)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3182) If server hosting META dies or is
stopping while processing another server shutdown, IOE accessing META stop
shutdown handler from finishing
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926811#action_12926811 ]
Jonathan Gray commented on HBASE-3182:
--------------------------------------
We need to do something here. First, I think we should do one of the wait-and-verify META methods. Even still, we could get an exception in MetaReader (here is another place we touch it).
In this case, we have no choice but to need to retry server shutdown or keep waiting. We needz meta.
> If server hosting META dies or is stopping while processing another server shutdown, IOE accessing META stop shutdown handler from finishing
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3182
> URL: https://issues.apache.org/jira/browse/HBASE-3182
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Reporter: Jonathan Gray
> Priority: Critical
> Fix For: 0.90.0
>
>
> In TestRollingRestart, there is a test which kills server hosting ROOT then immediately kills server hosting META. In a recent run this turned up a small race condition if the server hosting META is closing while we process shutdown of server hosting ROOT.
> {noformat}
> 2010-10-31 20:41:34,621 ERROR [MASTER_META_SERVER_OPERATIONS-dev692.sf2p.facebook.com:54989-0] executor.EventHandler(154): Caught throwable while processing event M_META_SERVER_SHUTDOWN
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Server not running
> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2216)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1652)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:749)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
> at $Proxy8.openScanner(Unknown Source)
> at org.apache.hadoop.hbase.catalog.MetaReader.getServerUserRegions(MetaReader.java:495)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:125)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3182) If server hosting META dies or is
stopping while processing another server shutdown, IOE accessing META stop
shutdown handler from finishing
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927128#action_12927128 ]
Jonathan Gray commented on HBASE-3182:
--------------------------------------
Tripped over this again but didn't even seem to relate to META. Just got this uncaught exception that killed my shutdown handler.
> If server hosting META dies or is stopping while processing another server shutdown, IOE accessing META stop shutdown handler from finishing
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3182
> URL: https://issues.apache.org/jira/browse/HBASE-3182
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Reporter: Jonathan Gray
> Priority: Critical
> Fix For: 0.90.0
>
>
> In TestRollingRestart, there is a test which kills server hosting ROOT then immediately kills server hosting META. In a recent run this turned up a small race condition if the server hosting META is closing while we process shutdown of server hosting ROOT.
> {noformat}
> 2010-10-31 20:41:34,621 ERROR [MASTER_META_SERVER_OPERATIONS-dev692.sf2p.facebook.com:54989-0] executor.EventHandler(154): Caught throwable while processing event M_META_SERVER_SHUTDOWN
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Server not running
> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2216)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1652)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:749)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
> at $Proxy8.openScanner(Unknown Source)
> at org.apache.hadoop.hbase.catalog.MetaReader.getServerUserRegions(MetaReader.java:495)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:125)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-3182) If server hosting META dies or is
stopping while processing another server shutdown, IOE accessing META stop
shutdown handler from finishing
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray resolved HBASE-3182.
----------------------------------
Resolution: Fixed
Fix for this was included in commit of HBASE-3181
> If server hosting META dies or is stopping while processing another server shutdown, IOE accessing META stop shutdown handler from finishing
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-3182
> URL: https://issues.apache.org/jira/browse/HBASE-3182
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 0.90.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Critical
> Fix For: 0.90.0
>
>
> In TestRollingRestart, there is a test which kills server hosting ROOT then immediately kills server hosting META. In a recent run this turned up a small race condition if the server hosting META is closing while we process shutdown of server hosting ROOT.
> {noformat}
> 2010-10-31 20:41:34,621 ERROR [MASTER_META_SERVER_OPERATIONS-dev692.sf2p.facebook.com:54989-0] executor.EventHandler(154): Caught throwable while processing event M_META_SERVER_SHUTDOWN
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Server not running
> at org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2216)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1652)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:749)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
> at $Proxy8.openScanner(Unknown Source)
> at org.apache.hadoop.hbase.catalog.MetaReader.getServerUserRegions(MetaReader.java:495)
> at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:125)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.