You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2018/04/27 21:49:00 UTC
[jira] [Comment Edited] (HBASE-20492) UnassignProcedure is stuck in
retry loop on region stuck in OPENING state
[ https://issues.apache.org/jira/browse/HBASE-20492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16455782#comment-16455782 ]
stack edited comment on HBASE-20492 at 4/27/18 9:48 PM:
--------------------------------------------------------
Made a subissue on how we can get into the retry loop. We can fix the found 'hole' but there are going to be others. The retries indicate something is wrong. The ever-cycling procedure, if there are more than one, can clog up the procedure executor but they flag that there is an issue. Not so good if many hosed regions. Need a back-off. Dropping the procedure probably ain't the best if it a sub-procedure of say, a macro-level disable table procedure. The disable could succeed though there are regions still assigned. Will be back.
was (Author: stack):
Made a subissue on how we can get into the retry loop. We can fix the found 'hole' but there are going to be others. The retries indicate something is wrong. The ever-cycling procedure, if there are more than one, can clog up the procedure executor but they flag that there is an issue. Not so good if many hosed regions. Need a back-off. Dropping the procedure probably ain't the best if it a sub-procedure of say, a disable. The disable could succeed though there are regions still assigned. Will be back.
> UnassignProcedure is stuck in retry loop on region stuck in OPENING state
> -------------------------------------------------------------------------
>
> Key: HBASE-20492
> URL: https://issues.apache.org/jira/browse/HBASE-20492
> Project: HBase
> Issue Type: Bug
> Components: amv2
> Affects Versions: 2.0.0
> Reporter: Umesh Agashe
> Priority: Critical
> Fix For: 2.0.1
>
>
> UnassignProcedure gets stuck in a retry loop for a region stuck in OPENING state. From logs:
> {code:java}
> 2018-04-25 15:59:53,825 WARN org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Retryable error trying to transition: pid=142564, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=IntegrationTestBigLinkedList_20180331004141, region=bd2fb2c7d39236c9b9085f350358df7c, server=vb1122.halxg.cloudera.com,22101,1522626198450; rit=OPENING, location=vb1122.halxg.cloudera.com,22101,1522626198450
> org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Expected [SPLITTING, SPLIT, MERGING, OPEN, CLOSING] so could move to CLOSING but current state=OPENING
> at org.apache.hadoop.hbase.master.assignment.RegionStates$RegionStateNode.transitionState(RegionStates.java:158)
> at org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1514)
> at org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:179)
> at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:309)
> at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:85)
> at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
> 2018-04-25 15:59:53,892 WARN org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: Retryable error trying to transition: pid=142564, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=IntegrationTestBigLinkedList_20180331004141, region=bd2fb2c7d39236c9b9085f350358df7c, server=vb1122.halxg.cloudera.com,22101,1522626198450; rit=OPENING, location=vb1122.halxg.cloudera.com,22101,1522626198450
> org.apache.hadoop.hbase.exceptions.UnexpectedStateException: Expected [SPLITTING, SPLIT, MERGING, OPEN, CLOSING] so could move to CLOSING but current state=OPENING
> at org.apache.hadoop.hbase.master.assignment.RegionStates$RegionStateNode.transitionState(RegionStates.java:158)
> at org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1514)
> at org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:179)
> at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:309)
> at org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:85)
> at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)