You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Bahram Chehrazy (JIRA)" <ji...@apache.org> on 2019/02/07 18:57:00 UTC
[jira] [Commented] (HBASE-21788) OpenRegionProcedure (after
recovery?) is unreliable and needs to be improved
[ https://issues.apache.org/jira/browse/HBASE-21788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16762966#comment-16762966 ]
Bahram Chehrazy commented on HBASE-21788:
-----------------------------------------
I have a similar repro of this. This time the meta is stuck in the OPENING state, but the server carrying the meta has already come on-line. The *OpenRegionProcedure* doesn't seem to be reporting back, because I don't see any more line with *pid=32701.*
2019-02-06 03:11:32,421 INFO [PEWorker-2] procedure.ServerCrashProcedure: Start *pid=32695*, state=RUNNABLE:*SERVER_CRASH_START*, hasLock=true; ServerCrashProcedure server=************,16020,1549448004110, splitWal=true, meta=true
2019-02-06 03:11:37,314 INFO [PEWorker-11] assignment.TransitRegionStateProcedure: Starting *pid=32700*, ppid=32695, state=RUNNABLE:REGION_*STATE_TRANSITION_GET_ASSIGN_CANDIDATE*, hasLock=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; rit=CLOSING, location=***********,16020,1549448004110; forceNewPlan=false, retain=false
2019-02-06 03:11:37,519 INFO [PEWorker-7] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as *<server1>,16020,1549450371876*
2019-02-06 03:11:37,525 INFO [PEWorker-7] procedure2.ProcedureExecutor: Initialized subprocedures=[\{*pid=32701*, *ppid=32700*, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment*.OpenRegionProcedure*}]
2019-02-06 03:11:39,728 WARN [master/*************:16000:becomeActiveMaster] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state=
{1588230740 state=*OPENING*, ts=1549451497525, server=*<server1>,16020,1549450371876*}
; ServerCrashProcedures=*true*. Master startup cannot progress, in holding-pattern until region onlined.
> OpenRegionProcedure (after recovery?) is unreliable and needs to be improved
> ----------------------------------------------------------------------------
>
> Key: HBASE-21788
> URL: https://issues.apache.org/jira/browse/HBASE-21788
> Project: HBase
> Issue Type: Bug
> Affects Versions: 3.0.0
> Reporter: Sergey Shelukhin
> Priority: Critical
>
> Not much for this one yet.
> I repeatedly see the cases when the region is stuck in OPENING, and after master restart RIT is recovered, and stays WAITING; its OpenRegionProcedure (also recovered) is stuck in Runnable and never does anything for hours. I cannot find logs on the target server indicating that it ever tried to do anything after master restart.
> This procedure needs at the very least logging of what it's trying to do, and maybe a timeout so it unconditionally fails after a configurable period (1 hour?).
> I may also investigate why it doesn't do anything and file a separate bug. I wonder if it's somehow related to the region status check, but this is just a hunch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)