You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2009/08/04 05:45:14 UTC
[jira] Created: (HBASE-1742) Region lost (disabled) when -ROOT-
offline
Region lost (disabled) when -ROOT- offline
------------------------------------------
Key: HBASE-1742
URL: https://issues.apache.org/jira/browse/HBASE-1742
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
I killed server with -ROOT-. It came back eventually but meantime we'd asked a server to open a region. It failed in the below but then we never try to open the region elsewhere.
{code}
803 2009-08-04 03:21:12,599 [regionserver/XX.XX.XX.140:60020.worker] ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: unable to process message: MSG_REGION_OPEN: TestTable,0916642860,1249356036404
804 java.lang.reflect.UndeclaredThrowableException
805 at $Proxy2.getRegionInfo(Unknown Source)
806 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:874)
807 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:515)
808 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)
809 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565)
810 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:524)
811 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)
812 at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:125)
813 at org.apache.hadoop.hbase.RegionHistorian.online(RegionHistorian.java:315)
814 at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1564)
815 at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1485)
816 at java.lang.Thread.run(Unknown Source)
817 Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
818 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2261)
819 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:1741)
820 at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
821 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
822 at java.lang.reflect.Method.invoke(Unknown Source)
823 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
824 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
825
826 at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:720)
827 at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:328)
828 ... 12 more
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1742) Region lost (disabled) when -ROOT-
offline or hosting server dies just before it tells master successful open
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1742:
-------------------------
Fix Version/s: (was: 0.20.1)
0.21.0
Moving to 0.21. These kinda conditions are what 0.21 is to address.
> Region lost (disabled) when -ROOT- offline or hosting server dies just before it tells master successful open
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1742
> URL: https://issues.apache.org/jira/browse/HBASE-1742
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.21.0
>
>
> I killed server with -ROOT-. It came back eventually but meantime we'd asked a server to open a region. It failed in the below but then we never try to open the region elsewhere.
> {code}
> 803 2009-08-04 03:21:12,599 [regionserver/XX.XX.XX.140:60020.worker] ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: unable to process message: MSG_REGION_OPEN: TestTable,0916642860,1249356036404
> 804 java.lang.reflect.UndeclaredThrowableException
> 805 at $Proxy2.getRegionInfo(Unknown Source)
> 806 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:874)
> 807 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:515)
> 808 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)
> 809 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565)
> 810 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:524)
> 811 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)
> 812 at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:125)
> 813 at org.apache.hadoop.hbase.RegionHistorian.online(RegionHistorian.java:315)
> 814 at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1564)
> 815 at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1485)
> 816 at java.lang.Thread.run(Unknown Source)
> 817 Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> 818 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2261)
> 819 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:1741)
> 820 at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> 821 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> 822 at java.lang.reflect.Method.invoke(Unknown Source)
> 823 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> 824 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
> 825
> 826 at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:720)
> 827 at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:328)
> 828 ... 12 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1742) Region lost (disabled) when -ROOT-
offline
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738795#action_12738795 ]
stack commented on HBASE-1742:
------------------------------
It seems to be stuck in the regionsintransition map with nothing checking for it being stuck there.
> Region lost (disabled) when -ROOT- offline
> ------------------------------------------
>
> Key: HBASE-1742
> URL: https://issues.apache.org/jira/browse/HBASE-1742
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> I killed server with -ROOT-. It came back eventually but meantime we'd asked a server to open a region. It failed in the below but then we never try to open the region elsewhere.
> {code}
> 803 2009-08-04 03:21:12,599 [regionserver/XX.XX.XX.140:60020.worker] ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: unable to process message: MSG_REGION_OPEN: TestTable,0916642860,1249356036404
> 804 java.lang.reflect.UndeclaredThrowableException
> 805 at $Proxy2.getRegionInfo(Unknown Source)
> 806 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:874)
> 807 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:515)
> 808 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)
> 809 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565)
> 810 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:524)
> 811 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)
> 812 at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:125)
> 813 at org.apache.hadoop.hbase.RegionHistorian.online(RegionHistorian.java:315)
> 814 at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1564)
> 815 at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1485)
> 816 at java.lang.Thread.run(Unknown Source)
> 817 Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> 818 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2261)
> 819 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:1741)
> 820 at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> 821 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> 822 at java.lang.reflect.Method.invoke(Unknown Source)
> 823 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> 824 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
> 825
> 826 at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:720)
> 827 at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:328)
> 828 ... 12 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1742) Region lost (disabled) when -ROOT-
offline or hosting server dies just before it tells master successful open
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1742:
-------------------------
Summary: Region lost (disabled) when -ROOT- offline or hosting server dies just before it tells master successful open (was: Region lost (disabled) when -ROOT- offline)
Found another case where we drop region.
HBASE-1743 lets me see into regions in transition. I can see the dropped regions stuck in the regionsInTransition map. Nothing is going to shift them. regionsInTransition needs to have a timer or be a queue or something so we look at its content from time to time and if stale, do something about it.
Its way too easy manufacturing these conditions. I ain't even doing the nasty stuff.
> Region lost (disabled) when -ROOT- offline or hosting server dies just before it tells master successful open
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1742
> URL: https://issues.apache.org/jira/browse/HBASE-1742
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> I killed server with -ROOT-. It came back eventually but meantime we'd asked a server to open a region. It failed in the below but then we never try to open the region elsewhere.
> {code}
> 803 2009-08-04 03:21:12,599 [regionserver/XX.XX.XX.140:60020.worker] ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: unable to process message: MSG_REGION_OPEN: TestTable,0916642860,1249356036404
> 804 java.lang.reflect.UndeclaredThrowableException
> 805 at $Proxy2.getRegionInfo(Unknown Source)
> 806 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:874)
> 807 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:515)
> 808 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)
> 809 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565)
> 810 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:524)
> 811 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)
> 812 at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:125)
> 813 at org.apache.hadoop.hbase.RegionHistorian.online(RegionHistorian.java:315)
> 814 at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1564)
> 815 at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1485)
> 816 at java.lang.Thread.run(Unknown Source)
> 817 Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> 818 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2261)
> 819 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:1741)
> 820 at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> 821 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> 822 at java.lang.reflect.Method.invoke(Unknown Source)
> 823 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> 824 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
> 825
> 826 at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:720)
> 827 at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:328)
> 828 ... 12 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1742) Region lost (disabled) when -ROOT-
offline or hosting server dies just before it tells master successful open
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1742:
-------------------------
Fix Version/s: 0.20.1
> Region lost (disabled) when -ROOT- offline or hosting server dies just before it tells master successful open
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1742
> URL: https://issues.apache.org/jira/browse/HBASE-1742
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.1
>
>
> I killed server with -ROOT-. It came back eventually but meantime we'd asked a server to open a region. It failed in the below but then we never try to open the region elsewhere.
> {code}
> 803 2009-08-04 03:21:12,599 [regionserver/XX.XX.XX.140:60020.worker] ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: unable to process message: MSG_REGION_OPEN: TestTable,0916642860,1249356036404
> 804 java.lang.reflect.UndeclaredThrowableException
> 805 at $Proxy2.getRegionInfo(Unknown Source)
> 806 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:874)
> 807 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:515)
> 808 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)
> 809 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565)
> 810 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:524)
> 811 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)
> 812 at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:125)
> 813 at org.apache.hadoop.hbase.RegionHistorian.online(RegionHistorian.java:315)
> 814 at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1564)
> 815 at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1485)
> 816 at java.lang.Thread.run(Unknown Source)
> 817 Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> 818 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2261)
> 819 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:1741)
> 820 at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> 821 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> 822 at java.lang.reflect.Method.invoke(Unknown Source)
> 823 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> 824 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
> 825
> 826 at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:720)
> 827 at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:328)
> 828 ... 12 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1742) Region lost (disabled) when -ROOT-
offline or hosting server dies just before it tells master successful open
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738854#action_12738854 ]
stack commented on HBASE-1742:
------------------------------
Yeah, bottom point is I can't even repair this table. Need to add to our 'tools' kicking region out of regionsInTransition before going on to mark it enabled, etc. Else, these operations have no effect.
> Region lost (disabled) when -ROOT- offline or hosting server dies just before it tells master successful open
> -------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-1742
> URL: https://issues.apache.org/jira/browse/HBASE-1742
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
>
> I killed server with -ROOT-. It came back eventually but meantime we'd asked a server to open a region. It failed in the below but then we never try to open the region elsewhere.
> {code}
> 803 2009-08-04 03:21:12,599 [regionserver/XX.XX.XX.140:60020.worker] ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: unable to process message: MSG_REGION_OPEN: TestTable,0916642860,1249356036404
> 804 java.lang.reflect.UndeclaredThrowableException
> 805 at $Proxy2.getRegionInfo(Unknown Source)
> 806 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:874)
> 807 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:515)
> 808 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)
> 809 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565)
> 810 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:524)
> 811 at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491)
> 812 at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:125)
> 813 at org.apache.hadoop.hbase.RegionHistorian.online(RegionHistorian.java:315)
> 814 at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1564)
> 815 at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1485)
> 816 at java.lang.Thread.run(Unknown Source)
> 817 Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> 818 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2261)
> 819 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:1741)
> 820 at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> 821 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> 822 at java.lang.reflect.Method.invoke(Unknown Source)
> 823 at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
> 824 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
> 825
> 826 at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:720)
> 827 at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:328)
> 828 ... 12 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.