[jira] [Resolved] (HBASE-30334) Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS

["Viraj Jasani (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-30334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Viraj Jasani resolved HBASE-30334.
----------------------------------
    Fix Version/s: 4.0.0-alpha-1
                   3.1.0
                   3.0.1
     Hadoop Flags: Reviewed
       Resolution: Fixed

> Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS
> -----------------------------------------------------------------------------
>
>                 Key: HBASE-30334
>                 URL: https://issues.apache.org/jira/browse/HBASE-30334
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 2.6.1, 2.5.10
>            Reporter: Nirdosh Kumar Yadav
>            Assignee: Nirdosh Kumar Yadav
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 4.0.0-alpha-1, 3.1.0, 3.0.1
>
>
> h4. {color:#172b4d}Problem Statement{color}
> {color:#172b4d}When MergeTableRegionsProcedure fails at the MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS state — e.g. because AssignmentManagerUtil.checkClosedRegion() throws "Recovered.edits are found in Region: … abort split/merge to prevent data loss" — the procedure framework unwinds through rollbackState(). The MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS case is a bare break;, so the parents that were CLOSED during the prior forward step MERGE_TABLE_REGIONS_CLOSE_REGIONS are never re-opened. Both parent regions remain OFFLINE and the RIT can persist indefinitely until an operator intervenes.{color}
>  
> {color:#172b4d}*What happened:*{color} * {color:#172b4d}Region {{112d9f08}} was gracefully moved from *rs-132 → rs-81* at {{{}16:36:34 UTC{}}}.{color}
>  * {color:#172b4d}The close on rs-132 had already flushed the region's edits, and rs-81 opened the region with {{{}openSeqNum=4997750282{}}}, establishing that the edits up to that point were durable.{color}
>  * {color:#172b4d}~25 seconds later, rs-132 was declared dead as part of the broader graceful RS drain.{color}
>  * {color:#172b4d}The WAL split worker (rs-34) subsequently created a {{recovered.edits}} file for this region containing edit {{{}seqId=4997750280{}}}.{color}
>  * {color:#172b4d}This edit was {*}already durable in the HFiles{*}. The recovered edit file was effectively stale/orphaned, but its existence was not recognized as harmless.{color}
> {color:#172b4d}*Where it got stuck:*{color} * {color:#172b4d}At {{{}18:32:33 UTC{}}}, {{MergeTableRegionsProcedure (pid=46990253)}} unassigned the region successfully.{color}
>  * {color:#172b4d}During {{{}MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS{}}}, the procedure saw the {{recovered.edits}} file and failed the check.{color}
>  * {color:#172b4d}The merge procedure retained the region lock and did not recover automatically, leaving the region in {*}CLOSED/RIT for ~48m 56s{*}.{color}
>  * {color:#172b4d}The RIT only cleared after the HMaster failover caused the procedure to replay and fresh top-level ASSIGN procedures reopened the regions.{color}
> h4. *Root Cause*
> MergeTableRegionsProcedure.rollbackState() at MergeTableRegionsProcedure.java:287-288:
>  
> case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS: break; // no-op — parents stay OFFLINE
> The symmetric case in SplitTableRegionProcedure.rollbackState() at SplitTableRegionProcedure.java:386-388 handles this correctly:
>  
> case SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS: openParentRegion(env); // calls reopenRegionsForRollback break;
> openParentRegion() (SplitTableRegionProcedure.java:644-648) delegates to the shared helper AssignmentManagerUtil.reopenRegionsForRollback() (AssignmentManagerUtil.java:283-290). The merge path is missing this call.
> h4. Key Observations 
>  * MergeTableRegionsProcedure pid=46990253 tripped checkClosedRegion at 18:32:33.293 UTC on parent 112d9f08181e80aaae01e68c9229c3cb.
>  * Rollback ran but did not re-open either parent (112d9f08… and 9e93ca76…), leaving both OFFLINE.
>  * The parents held an xlock preventing any fresh ASSIGN procedures from making progress.
>  * Stuck RIT lasted 48 minutes (18:32:33 → 19:21:29) on the OpenTSDB critical tier; only operator intervention (fresh ASSIGN procedures pid=47001645/47001646 after HMaster failover) cleared it.
>  
> The helper AssignmentManagerUtil.reopenRegionsForRollback() is already exercised by SplitTableRegionProcedure under production load, so risk is minimal. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)