You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "ramkrishna.s.vasudevan (Created) (JIRA)" <ji...@apache.org> on 2012/01/30 12:59:10 UTC
[jira] [Created] (HBASE-5299)
CatalogTracker.getMetaServerConnection() checks for root server connection
and makes waitForMeta to go into infinite loop in region assignment flow.
CatalogTracker.getMetaServerConnection() checks for root server connection and makes waitForMeta to go into infinite loop in region assignment flow.
----------------------------------------------------------------------------------------------------------------------------------------------------
Key: HBASE-5299
URL: https://issues.apache.org/jira/browse/HBASE-5299
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
RSA, RS B and RS C are 3 region servers.
RS A -> META
RS B -> ROOT
RS C -> NON META and NON ROOT
Kill RS B and wait for server shutdown handler to start.
Start RS B again before assigning ROOT to RS C.
Now the cluster will try to assign new regions to RS B.
But as ROOT is not yet assigned the OpenRegionHandler.updateMeta will fail to update the regions just because ROOT is not online.
{code}
a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2012-01-30 16:23:25,126 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Attempting to transition node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2012-01-30 16:23:25,159 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Successfully transitioned node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2012-01-30 16:23:35,385 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Attempting to transition node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2012-01-30 16:23:35,449 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Successfully transitioned node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2012-01-30 16:24:16,666 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Attempting to transition node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2012-01-30 16:24:16,701 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Successfully transitioned node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2012-01-30 16:24:20,788 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Interrupting thread Thread[PostOpenDeployTasks:a87109263ed53e67158377a149c5a7be,5,main]
2012-01-30 16:24:30,699 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=a87109263ed53e67158377a149c5a7be
org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Interrupted
at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:439)
at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1382)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:221)
{code}
So we need to wait for TM to assign the regions again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5299)
CatalogTracker.getMetaServerConnection() checks for root server connection
and makes waitForMeta to go into infinite loop in region assignment flow.
Posted by "stack (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13197362#comment-13197362 ]
stack commented on HBASE-5299:
------------------------------
I could take this one Ram. I'm trying to hack up testing harness for Master. Making a test for this condition would be a good exercise.
> CatalogTracker.getMetaServerConnection() checks for root server connection and makes waitForMeta to go into infinite loop in region assignment flow.
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5299
> URL: https://issues.apache.org/jira/browse/HBASE-5299
> Project: HBase
> Issue Type: Bug
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Priority: Minor
>
> RSA, RS B and RS C are 3 region servers.
> RS A -> META
> RS B -> ROOT
> RS C -> NON META and NON ROOT
> Kill RS B and wait for server shutdown handler to start.
> Start RS B again before assigning ROOT to RS C.
> Now the cluster will try to assign new regions to RS B.
> But as ROOT is not yet assigned the OpenRegionHandler.updateMeta will fail to update the regions just because ROOT is not online.
> {code}
> a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:23:25,126 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Attempting to transition node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:23:25,159 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Successfully transitioned node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:23:35,385 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Attempting to transition node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:23:35,449 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Successfully transitioned node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:24:16,666 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Attempting to transition node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:24:16,701 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Successfully transitioned node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:24:20,788 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Interrupting thread Thread[PostOpenDeployTasks:a87109263ed53e67158377a149c5a7be,5,main]
> 2012-01-30 16:24:30,699 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=a87109263ed53e67158377a149c5a7be
> org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Interrupted
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:439)
> at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1382)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:221)
> {code}
> So we need to wait for TM to assign the regions again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5299)
CatalogTracker.getMetaServerConnection() checks for root server connection
and makes waitForMeta to go into infinite wait in region assignment flow.
Posted by "ramkrishna.s.vasudevan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ramkrishna.s.vasudevan updated HBASE-5299:
------------------------------------------
Summary: CatalogTracker.getMetaServerConnection() checks for root server connection and makes waitForMeta to go into infinite wait in region assignment flow. (was: CatalogTracker.getMetaServerConnection() checks for root server connection and makes waitForMeta to go into infinite loop in region assignment flow.)
> CatalogTracker.getMetaServerConnection() checks for root server connection and makes waitForMeta to go into infinite wait in region assignment flow.
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-5299
> URL: https://issues.apache.org/jira/browse/HBASE-5299
> Project: HBase
> Issue Type: Bug
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Priority: Minor
>
> RSA, RS B and RS C are 3 region servers.
> RS A -> META
> RS B -> ROOT
> RS C -> NON META and NON ROOT
> Kill RS B and wait for server shutdown handler to start.
> Start RS B again before assigning ROOT to RS C.
> Now the cluster will try to assign new regions to RS B.
> But as ROOT is not yet assigned the OpenRegionHandler.updateMeta will fail to update the regions just because ROOT is not online.
> {code}
> a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:23:25,126 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Attempting to transition node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:23:25,159 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Successfully transitioned node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:23:35,385 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Attempting to transition node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:23:35,449 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Successfully transitioned node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:24:16,666 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Attempting to transition node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:24:16,701 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1352e27539c0009 Successfully transitioned node a87109263ed53e67158377a149c5a7be from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
> 2012-01-30 16:24:20,788 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Interrupting thread Thread[PostOpenDeployTasks:a87109263ed53e67158377a149c5a7be,5,main]
> 2012-01-30 16:24:30,699 WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Exception running postOpenDeployTasks; region=a87109263ed53e67158377a149c5a7be
> org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Interrupted
> at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:439)
> at org.apache.hadoop.hbase.catalog.MetaEditor.updateRegionLocation(MetaEditor.java:142)
> at org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:1382)
> at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:221)
> {code}
> So we need to wait for TM to assign the regions again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira