You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2010/11/01 23:40:27 UTC
[jira] Created: (HBASE-3187) RS stuck waiting on meta, doesn't
recover
RS stuck waiting on meta, doesn't recover
-----------------------------------------
Key: HBASE-3187
URL: https://issues.apache.org/jira/browse/HBASE-3187
Project: HBase
Issue Type: Bug
Reporter: stack
Fix For: 0.90.0
Saw this testing.... All opener handlers stuck waiting on meta... doesn't recover... This RS had also been given a bunch of regions to open.
{code}
"RS_OPEN_REGION-sv2borg188,60020,1288644070661-2" daemon prio=10 tid=0x0000000042c65000 nid=0x4e61 in Object.wait() [0x00007f9709e5b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:327)
- locked <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:362)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:146)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1280)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:156)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"RS_OPEN_REGION-sv2borg188,60020,1288644070661-1" daemon prio=10 tid=0x0000000042ce7800 nid=0x4e5c in Object.wait() [0x00007f9709f5c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:327)
- locked <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:362)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:146)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1280)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:156)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3187) RS stuck waiting on meta, doesn't
recover
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929686#action_12929686 ]
Jonathan Gray commented on HBASE-3187:
--------------------------------------
So then where is bug? Shouldn't these guys be waiting indefinitely for availability of META? Bug is only if these guys are somehow blocking opening or META?
> RS stuck waiting on meta, doesn't recover
> -----------------------------------------
>
> Key: HBASE-3187
> URL: https://issues.apache.org/jira/browse/HBASE-3187
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.90.0
>
>
> Saw this testing.... All opener handlers stuck waiting on meta... doesn't recover... This RS had also been given a bunch of regions to open.
> {code}
> "RS_OPEN_REGION-sv2borg188,60020,1288644070661-2" daemon prio=10 tid=0x0000000042c65000 nid=0x4e61 in Object.wait() [0x00007f9709e5b000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:327)
> - locked <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:362)
> at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:146)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1280)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:156)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> "RS_OPEN_REGION-sv2borg188,60020,1288644070661-1" daemon prio=10 tid=0x0000000042ce7800 nid=0x4e5c in Object.wait() [0x00007f9709f5c000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:327)
> - locked <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:362)
> at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:146)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1280)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:156)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-3187) RS stuck waiting on meta, doesn't
recover
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-3187.
--------------------------
Resolution: Invalid
The bug is that all openers are blocked waiting on a meta that was not going to open or that was open.
I'm going to close this bug as ill-defined. I failed to capture why this was a problem and fail to remember the circumstance.
It'll happen again if it wants to be fixed.
> RS stuck waiting on meta, doesn't recover
> -----------------------------------------
>
> Key: HBASE-3187
> URL: https://issues.apache.org/jira/browse/HBASE-3187
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.90.0
>
>
> Saw this testing.... All opener handlers stuck waiting on meta... doesn't recover... This RS had also been given a bunch of regions to open.
> {code}
> "RS_OPEN_REGION-sv2borg188,60020,1288644070661-2" daemon prio=10 tid=0x0000000042c65000 nid=0x4e61 in Object.wait() [0x00007f9709e5b000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:327)
> - locked <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:362)
> at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:146)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1280)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:156)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> "RS_OPEN_REGION-sv2borg188,60020,1288644070661-1" daemon prio=10 tid=0x0000000042ce7800 nid=0x4e5c in Object.wait() [0x00007f9709f5c000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:327)
> - locked <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:362)
> at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:146)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1280)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:156)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3187) RS stuck waiting on meta, doesn't
recover
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929685#action_12929685 ]
stack commented on HBASE-3187:
------------------------------
Yeah, we have separate handlers for .META. but this is ordinary open handler needing to update .META. w/ new location. Don't know where the .META. was at time.
> RS stuck waiting on meta, doesn't recover
> -----------------------------------------
>
> Key: HBASE-3187
> URL: https://issues.apache.org/jira/browse/HBASE-3187
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.90.0
>
>
> Saw this testing.... All opener handlers stuck waiting on meta... doesn't recover... This RS had also been given a bunch of regions to open.
> {code}
> "RS_OPEN_REGION-sv2borg188,60020,1288644070661-2" daemon prio=10 tid=0x0000000042c65000 nid=0x4e61 in Object.wait() [0x00007f9709e5b000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:327)
> - locked <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:362)
> at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:146)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1280)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:156)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> "RS_OPEN_REGION-sv2borg188,60020,1288644070661-1" daemon prio=10 tid=0x0000000042ce7800 nid=0x4e5c in Object.wait() [0x00007f9709f5c000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:327)
> - locked <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:362)
> at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:146)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1280)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:156)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3187) RS stuck waiting on meta, doesn't
recover
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929642#action_12929642 ]
Jonathan Gray commented on HBASE-3187:
--------------------------------------
Don't we have separate handler for opening root/meta? Where was META on this cluster?
> RS stuck waiting on meta, doesn't recover
> -----------------------------------------
>
> Key: HBASE-3187
> URL: https://issues.apache.org/jira/browse/HBASE-3187
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.90.0
>
>
> Saw this testing.... All opener handlers stuck waiting on meta... doesn't recover... This RS had also been given a bunch of regions to open.
> {code}
> "RS_OPEN_REGION-sv2borg188,60020,1288644070661-2" daemon prio=10 tid=0x0000000042c65000 nid=0x4e61 in Object.wait() [0x00007f9709e5b000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:327)
> - locked <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:362)
> at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:146)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1280)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:156)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> "RS_OPEN_REGION-sv2borg188,60020,1288644070661-1" daemon prio=10 tid=0x0000000042ce7800 nid=0x4e5c in Object.wait() [0x00007f9709f5c000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:327)
> - locked <0x00007f972268ad00> (a java.util.concurrent.atomic.AtomicBoolean)
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:362)
> at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:146)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1280)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:156)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.