You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2009/01/11 23:44:59 UTC
[jira] Created: (HBASE-1121) Cluster confused about where -ROOT- is
Cluster confused about where -ROOT- is
--------------------------------------
Key: HBASE-1121
URL: https://issues.apache.org/jira/browse/HBASE-1121
Project: Hadoop HBase
Issue Type: Bug
Environment: TRUNK trying to do big upload
Reporter: stack
Each node in cluster is doing below:
{code}
2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
>. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
{code}
I tried to close the root region and tripped over hbase-1120.
So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663239#action_12663239 ]
stack commented on HBASE-1121:
------------------------------
hbase-1123 killed investigation of this issue. I committed the better logging for next time.
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Attachments: 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman reassigned HBASE-1121:
------------------------------------
Assignee: Jim Kellerman
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Assignee: Jim Kellerman
> Attachments: 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12666952#action_12666952 ]
Jim Kellerman commented on HBASE-1121:
--------------------------------------
> Andrew Purtell - 24/Jan/09 04:03 AM
> Did you change how "hbase.regionserver.safemode.period" is handled on the HRS?
Yes. The HRS now waits for the master to tell it when safe mode is off.
> Does the HRS still run the safe mode thread for slowly ramping up compactions/splits on startup?
Yes. After exiting safe mode, HRS slowly ramps up compactions and splits. The thread is now
called the CompactionLimitThread
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Assignee: Jim Kellerman
> Fix For: 0.19.1, 0.20.0
>
> Attachments: 1121.patch, 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665623#action_12665623 ]
stack commented on HBASE-1121:
------------------------------
The scheme for setting root region in HRS looks badly broken. Its set once on line #315 and the flat haveRootRegion makes it so its never ever reset.
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Assignee: Jim Kellerman
> Attachments: 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662840#action_12662840 ]
stack commented on HBASE-1121:
------------------------------
It looks like its the regionhistorian trying to update about a compaction that is failing:
{code}
2009-01-11 06:11:56,405 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.regionserver.HStore: moving /hbasetrunk2/TestTable/compaction.dir/1701213566/info/mapfiles/4916277877406001197 to /hbasetrunk2/TestTable/1701213566/info/mapfiles/3125166216229916041
2009-01-11 06:11:56,472 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 1701213566/info store size is 188.2m
2009-01-11 06:11:56,480 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region TestTable,0008454525,1231561159116 in 1mins, 8sec
2009-01-11 06:11:56,484 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
>. Retrying after sleep of 2000
2009-01-11 06:11:56,487 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}
2009-01-11 06:11:58,491 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
>. Retrying after sleep of 2000
...
{code}
Above goes on till attempt 8, then it starts over again at 0 and just keeps on going.
Why ain't we recalibrating to find new -ROOT- location.
Cluster is hosed unless I restart a regionserver one at a time or just restart whole cluster.
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663038#action_12663038 ]
stack commented on HBASE-1121:
------------------------------
This issue has aspects of HBASE-964
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12666707#action_12666707 ]
Jim Kellerman commented on HBASE-1121:
--------------------------------------
Summary of changes:
hbase-default.xml:
- change default HLog block size from 64MB to 1MB
- remove hbase.regionserver.safemode.period - not in any way related to actual cluster safe mode. Now HRegionServer gets safe mode status from master.
src/test/hbase-site.xml:
- disable safe mode for regression tests. Necessary for them to run successfully
src/test/TestSerialization:
- add missing @Override annotations
- since some statics have been moved out of HLog, use constructor to create HMsg objects
HRegionServer:
- move some statics from HLog to here because that is the only place they are used
- rename SafeModeThread to CompactionLimitThread because that it what it was really doing.
- CompactionLimitThread now waits for the cluster to exit safe mode rather than waiting an arbitrary period of time
- if in safe mode, look at HMsg from master to determine if safe mode is done
HLog:
- Change default block size from 64MB to 1MB
- Remove statics used only by a single class
- Add new constructors so that Master can tell region server if safe mode is still in effect
ServerManager:
- Move some of the HLog statics here.
- use new HLog constructors to tell region server if safe mode is still in effect
- only cache root region location during safe mode. Once we exit safe mode we unset it so that HConnectionManager will recalibrate if the root region location changes.
ProcessRegionOpen, ProcessRegionClose
- metaRegion is now variable and these classes need to call their parent class to determine the location of the meta region they need. Previously, if the meta region moved, these classes would not recalibrate.
RegionManager:
- use new HMsg constructors to tell region servers whether we are in safe mode or not.
- once we exit safe mode, we unset the root region location in HConnectionManager so that it will recalibrate if the root region moves.
ProcessRegionStatusChange:
- parent class of ProcessRegionOpen and ProcessRegionClose
- implements getMetaRegion which provides for reassignment of meta regions from one server to another
HConnectionManager:
- do not cache root region location. If recalibrating, ask the master where the root region is.
HMsg:
- added private boolean safeMode (settable only by master)
- Move static declarations to modules that use them
- Add new constructors so master can inform region servers of safe mode status
- add new accessor method (isInSafeMode()) so region servers can determine if safe mode is still in effect
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Assignee: Jim Kellerman
> Attachments: 1121.patch, 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662842#action_12662842 ]
stack commented on HBASE-1121:
------------------------------
This looks like where things started to go wrong:
{code}
2009-01-11 06:11:13,560 [IPC Server handler 0 on 60020] WARN org.apache.hadoop.ipc.HBaseServer: IPC Server Responder, call batchUpdates([B@18e6b7f2, [Lorg.apache.hadoop.hbase.io.BatchUpdate;@6ff74992) from XX.XX.XX.141:40706: output error
2009-01-11 06:11:13,561 [IPC Server handler 0 on 60020] INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020 caught: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1084)
at org.apache.hadoop.hbase.ipc.HBaseServer.access$1900(HBaseServer.java:70)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:592)
at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:656)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:922)
{code}
141 went away.
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1121) Cluster confused about where -ROOT- is
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1121:
-------------------------
Attachment: 1121.patch
Better logging in HCM so can try to tell whats happening around rediscovery of -ROOT- location
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Attachments: 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665626#action_12665626 ]
Jim Kellerman commented on HBASE-1121:
--------------------------------------
The reason for this is that the root region needs to be set before any HTable accesses are
done. It is never reset, because it is assumed that HTable/HConnectionManager will detect if
the root region has moved and recalibrate accordingly. During the startup process, an HTable
in a server process will never start because it cannot find the root or meta regions (because
it is trying to serve the root or meta regions). It is a total hack to work around the fact
that HTable/HConnectionManager do not work in server processes, especially during safe-mode.
This is why I have advocated in the past for never using HTable in server processes. The only
reason we have done so, is that doing HRegionServer RPC's for similar HTable requests are a
whole lot more complicated, and would require a duplication of code (or a significant
refactoring - I am opposed to duplication of code, but the refactoring is not going to be
trivial - in fact, I see it as being very difficult)
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Assignee: Jim Kellerman
> Attachments: 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663151#action_12663151 ]
stack commented on HBASE-1121:
------------------------------
So its not 'innocuous' as thought above in that we just lose the regionhistorian updates. It actually makes compactions/splits fail:
{code}
ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction/Split failed for region...
{code}
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665985#action_12665985 ]
Jim Kellerman commented on HBASE-1121:
--------------------------------------
I managed to reproduce this on a three node cluster as follows:
- ensure root and meta regions are assigned to different nodes
- while running PerformanceEvaluation
-- first kill the root server (and wait for it to be reassigned) - it needs to land on the node that is not serving meta
-- kill the server hosting meta
the remaining server never refinds root or meta.
The patch does not help if the root lands on the same node as the master.
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Assignee: Jim Kellerman
> Attachments: 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1121) Cluster confused about where -ROOT- is
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-1121:
---------------------------------
Attachment: 1121.patch
Patch for trunk.
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Assignee: Jim Kellerman
> Attachments: 1121.patch, 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665742#action_12665742 ]
stack commented on HBASE-1121:
------------------------------
I've spent a bunch of time trying to reproduce but cannot. I put up 4 node cluster and tried various sequences of moving -ROOT- around cluster and then .META. I wanted to reproduce so I could prove the J-D (+nitay) found issue fixed this prob.
I think I should commit the below anyways:
{code}
Index: src/java/org/apache/hadoop/hbase/client/HConnectionManager.java
===================================================================
--- src/java/org/apache/hadoop/hbase/client/HConnectionManager.java (revision 736178)
+++ src/java/org/apache/hadoop/hbase/client/HConnectionManager.java (working copy)
@@ -445,7 +445,8 @@
// second waits. The second thread will not do find.
if (!useCache || rootRegionLocation == null) {
- return locateRootRegion();
+ // Cache new location.
+ this.rootRegionLocation = locateRootRegion();
}
r
{code}
...maybe I need to open a new issue to do it.
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Assignee: Jim Kellerman
> Attachments: 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662836#action_12662836 ]
stack commented on HBASE-1121:
------------------------------
This is on one of the regionservers that had been hosting root... 139... its being asked for the -ROOT- by 140 though its been restarted.
{code}2009-01-11 23:04:39,674 [IPC Server handler 0 on 60020] ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
2009-01-11 23:04:39,674 [IPC Server handler 0 on 60020] INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020, call getClosestRowBefore([B@32d27bb0, [B@18b679e1, [B@73fbaf73) from XX.XX.XX.140:53563: error: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894){code}
On 140, 142 thinks it has the -ROOT-:
{code}
2009-01-11 23:22:27,344 [IPC Server handler 4 on 60020] INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60020, call getClosestRowBefore([B@10321250, [B@47504a7e, [B@684abac5) from XX.XX.XX.142:58471: error: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
{code}
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman resolved HBASE-1121.
----------------------------------
Resolution: Fixed
Fix Version/s: 0.20.0
0.19.1
Committed to branch and trunk.
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Assignee: Jim Kellerman
> Fix For: 0.19.1, 0.20.0
>
> Attachments: 1121.patch, 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12666899#action_12666899 ]
Andrew Purtell commented on HBASE-1121:
---------------------------------------
Did you change how "hbase.regionserver.safemode.period" is handled on the HRS? Does the HRS still run the safe mode thread for slowly ramping up compactions/splits on startup?
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Assignee: Jim Kellerman
> Fix For: 0.19.1, 0.20.0
>
> Attachments: 1121.patch, 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662833#action_12662833 ]
stack commented on HBASE-1121:
------------------------------
So, clients get right answer when they ask for where -ROOT- is. UI is all up running w/ -ROOT- in same place. The restarted regionserver complains about the -ROOT- location wth above exception.
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665954#action_12665954 ]
stack commented on HBASE-1121:
------------------------------
JK committed above. He doesn't think it'll do any harm.
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Assignee: Jim Kellerman
> Attachments: 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662848#action_12662848 ]
stack commented on HBASE-1121:
------------------------------
... will see this:
{code}
2009-01-11 23:47:26,255 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}
2009-01-11 23:49:02,268 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] WARN org.apache.hadoop.hbase.RegionHistorian: Unable to 'Region compaction completed in 10sec'
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server Some server for region , row 'TestTable,0000442359,1231570256381', but failed after 11 attempts.
Exceptions:
{code}
They might be 'harmless' in that it happens every time the regionhistorian tries to report a new event. Really noisy though.
My cluster is broke. All of the above makes it hard to debug.
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1121) Cluster confused about where -ROOT-
is
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12665632#action_12665632 ]
stack commented on HBASE-1121:
------------------------------
If HTable in servers is a 'total hack' we should move them out.
J-D (or Nitay) have found why HCM doesn't recalibrate:
{code}
if (!useCache || rootRegionLocation == null) {
return locateRootRegion();
}
return rootRegionLocation;
{code}
rootRegionLocation is not set with the just found location.
Going to test locally.
> Cluster confused about where -ROOT- is
> --------------------------------------
>
> Key: HBASE-1121
> URL: https://issues.apache.org/jira/browse/HBASE-1121
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK trying to do big upload
> Reporter: stack
> Assignee: Jim Kellerman
> Attachments: 1121.patch
>
>
> Each node in cluster is doing below:
> {code}
> 2009-01-11 20:52:22,739 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> >. Retrying after sleep of 20002009-01-11 20:52:22,742 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Found ROOT REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', COMPRESSION => 'NONE', VERSIONS => '10', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'true'}], INDEXES => []}}2009-01-11 20:52:24,759 [regionserver/0:0:0:0:0:0:0:0:60020.compactor] DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 1 of 10 failed with <org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2064)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1545)
> at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:894)
> {code}
> I tried to close the root region and tripped over hbase-1120.
> So I killed the regionserver hosting -ROOT- but weirdly had no effect as though the regionservers' internal cache of -ROOT- location cannot be changed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.