You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by bhupendra jain <bh...@huawei.com> on 2016/06/21 14:37:00 UTC
Not able to enable Table after network error
Hi all
We encountered a strange scenario in our Hbase cluster ( based on 1.0 branch). Scenario is like below
There is a table (t1) with region (r1) in disabled state. Region r1 was last assigned to Region Server (RS1). For some duration in between, Network communication got broken between HMaster (HM1) and RegionServer (RS1).
In this duration , when user tries to enable table t1, it failed. This happened because the region r1 couldn't assign to any of the live RS. The assignment got skipped form forceRegionStateToOffline() method in AssignmentMaanger due to below check
if (useZKForAssignment
&& regionStates.isServerDeadAndNotProcessed(sn)
&& wasRegionOnDeadServerByMeta(region, sn)) {
}
We found that the method regionStates.isServerDeadAndNotProcessed(sn), will put the RS1 in its deadServers and wait for SSH to process the RS1 which never happens as session between RS1 and ZK is still fine.
synchronized boolean isServerDeadAndNotProcessed(ServerName server) {
-----
if (serverManager.isServerReachable(server)) {
return false;
}
// The size of deadServers won't grow unbounded.
deadServers.put(hostAndPort, Long.valueOf(startCode));
}
// Watch out! If the server is not dead, the region could
// remain unassigned. That's why ServerManager#isServerReachable
// should use some retry.
-----
Even though Network recovered after some time, The table could not be enabled after that. Its due to
a) deadServers never removes the entry of RS1
b) Even though entry from deadServers is removed OR RS is aborted, the table cannot be enabled as its already in EANBLING state. Only when Master failover happens, the table gets enabled.
Similar scenario is also discussed in below JIRAs
https://issues.apache.org/jira/browse/HBASE-9514?focusedCommentId=13769761&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13769761
https://issues.apache.org/jira/browse/HBASE-6469
Let us know how to handle this scenario OR any other mechanism.
Thanks
Bhupendra
--------------------------------------------------------------------------------------------------------
This e-mail and its attachments contain confidential information from HUAWEI, which
is intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!