You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Andrew Kyle Purtell (Jira)" <ji...@apache.org> on 2022/06/16 18:13:00 UTC
[jira] [Resolved] (HBASE-9665) Region gets lost when balancer & SSH both trying to assign
[ https://issues.apache.org/jira/browse/HBASE-9665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Kyle Purtell resolved HBASE-9665.
----------------------------------------
Resolution: Incomplete
> Region gets lost when balancer & SSH both trying to assign
> -----------------------------------------------------------
>
> Key: HBASE-9665
> URL: https://issues.apache.org/jira/browse/HBASE-9665
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
> Affects Versions: 0.96.0
> Reporter: Jeffrey Zhong
> Priority: Critical
>
> In summary, a server dies and its regions are re-assigned. While right before SSH, balancer is starting assign one region on the server to somewhere.
> The balancer assignment got preempted by the SSH assignment:
> {code}
> 2013-09-25 11:55:32,854 INFO Priority.RpcServer.handler=7,port=60020 regionserver.HRegionServer: Received CLOSE for the region:6deb1bfefe8cbdb443084efe919fdeb7 , which we are already trying to OPEN. Cancelling OPENING.
> {code}
> The SSH assignment(by GeneralBulkAssigner) failed too due to:
> {code}
> 2013-09-25 11:55:32,927 WARN [RS_OPEN_REGION-hor15n09:60020-2] zookeeper.ZKAssign: regionserver:60020-0x14153d449d30ad0 Attempt to transition the unassigned node for 6deb1bfefe8cbdb443084efe919fdeb7 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING failed, the server that tried to transition was hor15n09.gq1.ygridcore.net,60020,1380109280320 not the expected hor15n07.gq1.ygridcore.net,60020,1380109890414
> {code}
> In the end, the region 6deb1bfefe8cbdb443084efe919fdeb7 is lost.
> Below is the master log, you can see both balancer and SSH try to assign the region around the same time:
> {code}
> 2013-09-25 11:55:32,731 INFO [MASTER_SERVER_OPERATIONS-hor15n05:60000-4] master.RegionStates: Transitioning {6deb1bfefe8cbdb443084efe919fdeb7 state=PENDING_CLOSE, ts=1380110132710, server=hor15n12.gq1.ygridcore.net,60020,1380109596307} will be handled by SSH for hor15n12.gq1.ygridcore.net,60020,1380109596307
> ...
> 2013-09-25 11:55:32,849 INFO [hor15n05.gq1.ygridcore.net,60000,1380108611483-BalancerChore] master.RegionStates: Transitioned {6deb1bfefe8cbdb443084efe919fdeb7 state=OFFLINE, ts=1380110132768, server=null} to {6deb1bfefe8cbdb443084efe919fdeb7 state=PENDING_OPEN, ts=1380110132849, server=hor15n07.gq1.ygridcore.net,60020,1380109890414}
> ...
> 2013-09-25 11:55:32,898 INFO [hor15n05.gq1.ygridcore.net,60000,1380108611483-GeneralBulkAssigner-1] master.RegionStates: Transitioned {6deb1bfefe8cbdb443084efe919fdeb7 state=OFFLINE, ts=1380110132861, server=null} to {6deb1bfefe8cbdb443084efe919fdeb7 state=PENDING_OPEN, ts=1380110132898, server=hor15n09.gq1.ygridcore.net,60020,1380109280320}
> {code}
> Since SSH force region assignment while it doesn't recreate offline znode, the later region opening would fail with the following error. I'm suggesting to recreate offline znode when we force a region assignment(forceNewPlan=true) with low impact.
> {code}
> 2013-09-25 11:55:32,927 WARN [RS_OPEN_REGION-hor15n09:60020-2] zookeeper.ZKAssign: regionserver:60020-0x14153d449d30ad0 Attempt to transition the unassigned node for 6deb1bfefe8cbdb443084efe919fdeb7 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING failed, the server that tried to transition was hor15n09.gq1.ygridcore.net,60020,1380109280320 not the expected hor15n07.gq1.ygridcore.net,60020,1380109890414
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)