You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Duo Zhang (Jira)" <ji...@apache.org> on 2022/04/09 08:14:00 UTC
[jira] [Commented] (HBASE-26914) Rebuild a cluster from an existing root directory may hang
[ https://issues.apache.org/jira/browse/HBASE-26914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17519926#comment-17519926 ]
Duo Zhang commented on HBASE-26914:
-----------------------------------
The behavior is expected. It will try to assign the region again. Later after we schedule a SCP, the SCP will interrupt this TRSP and let it assign the region to another region server.
> Rebuild a cluster from an existing root directory may hang
> ----------------------------------------------------------
>
> Key: HBASE-26914
> URL: https://issues.apache.org/jira/browse/HBASE-26914
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 3.0.0-alpha-2
> Reporter: LiangJun He
> Assignee: LiangJun He
> Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> After HBASE-26245, we can rebuild a cluster from an existing root directory on cloud environment, but we still have a problem, if we stop the regionservers of the old cluster first, and then stop the masters, new cluster(rebuild from old cluster's root directory) may hang.
> This problem is also described in HBASE-26898.
>
> Hang Reason:
> For example, if RS1 is stopped first, the Master will expire RS1, and will generate a ServerCrashProcedure to reassign the region on RS1 (at the same time, the Master will delete the RS1 node information saved in the local region), and then generate subProcedure TransitRegionStateProcedure for reassignment, that is planned to assign the region to RS2 (RS2 has not been stopped at this time), but then RS2 is stopped, and the Master is also stopped.
> Rebuild a cluster from an existing root directory(from old cluster on OSS), the Master will recover from the Procedure state and continue to execute the unfinished Procedure. At this time, there is an uncompleted TransitRegionStateProcedure, and the target-RS to be assigned to the region is the RS2 of the old cluster, it will report warn message：
> {code:java}
> 2022-04-09 14:44:00,050 INFO [master/emr-header-1:16000:becomeActiveMaster] procedure.MasterProcedureScheduler: Took xlock for pid=33950, ppid=33615, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=false; TransitRegionStateProcedure table=usertable, region=5c0dac3f7410e8c91e00bcfcdc6e774a, ASSIGN
> 2022-04-09 14:44:00,241 INFO [master/emr-header-1:16000:becomeActiveMaster] assignment.AssignmentManager: Attach pid=33950, ppid=33615, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=false; TransitRegionStateProcedure table=usertable, region=5c0dac3f7410e8c91e00bcfcdc6e774a, ASSIGN to state=OFFLINE, location=null, table=usertable, region=5c0dac3f7410e8c91e00bcfcdc6e774a to restore RIT
> 2022-04-09 14:44:11,678 WARN [PEWorker-1] assignment.RegionRemoteProcedureBase: Can not add remote operation pid=35353, ppid=33950, state=RUNNABLE, hasLock=true; OpenRegionProcedure 5c0dac3f7410e8c91e00b
> cfcdc6e774a, server=emr-worker-1.cluster-18871,16020,1649346155842 for region {ENCODED => 5c0dac3f7410e8c91e00bcfcdc6e774a, NAME => 'usertable,user1004,1649256577246.5c0dac3f7410e8c91e00bcfcdc6e774a.', ST
> ARTKEY => 'user1004', ENDKEY => 'user1008'} to server emr-worker-1.cluster-18871,16020,1649346155842, this usually because the server is alread dead, give up and mark the procedure as complete, the parent
> procedure will take care of this.
> org.apache.hadoop.hbase.procedure2.NoServerDispatchException: emr-worker-1.cluster-18871,16020,1649346155842; pid=35353, ppid=33950, state=RUNNABLE, hasLock=true; OpenRegionProcedure 5c0dac3f7410e8c91e00b
> cfcdc6e774a, server=emr-worker-1.cluster-18871,16020,1649346155842
> at org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher.addOperationToNode(RemoteProcedureDispatcher.java:172)
> at org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:283)
> at org.apache.hadoop.hbase.master.assignment.RegionRemoteProcedureBase.execute(RegionRemoteProcedureBase.java:56)
> at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:953)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1667)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1414)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:78)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1981)
> 2022-04-09 14:44:11,689 INFO [PEWorker-1] procedure2.ProcedureExecutor: Finished pid=35353, ppid=33950, state=SUCCESS, hasLock=false; OpenRegionProcedure 5c0dac3f7410e8c91e00bcfcdc6e774a, server=emr-worker-1.cluster-18871,16020,1649346155842 in 2 mins, 37.672 sec
> 2022-04-09 14:44:11,697 INFO [PEWorker-1] procedure2.ProcedureExecutor: Finished pid=33950, ppid=33615, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=usertable, region=5c0dac3f7410e8c91e00bcfcdc6e774a, ASSIGN in 2 mins, 38.865 sec{code}
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)