You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "nirdosh0110 (via GitHub)" <gi...@apache.org> on 2026/08/28 09:35:25 UTC
[PR] HBASE-30334 Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS [hbase]
nirdosh0110 opened a new pull request, #8583:
URL: https://github.com/apache/hbase/pull/8583
## Summary
- The `MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS` case in `MergeTableRegionsProcedure.rollbackState()` was a bare `break;`. When rollback fires at this step (e.g. `checkClosedRegions` fails), the parent regions stay CLOSED and produce an indefinite stuck RIT with no automatic recovery.
- Fall through `CHECK_CLOSED_REGIONS` to `CLOSE_REGIONS` so the existing `rollbackCloseRegionsForMerge` helper reopens the parents via `AssignmentManagerUtil.reopenRegionsForRollback`.
- Mirrors the analogous `SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS` handling in `SplitTableRegionProcedure.rollbackState()` (which calls `openParentRegion(env)`).
JIRA: https://issues.apache.org/jira/browse/HBASE-30334
## Test plan
- [ ] Existing rollback tests in `TestMergeTableRegionsProcedure` still pass.
- [ ] Manual/soak: force `checkClosedRegions` to throw in a merge procedure and confirm parents get reopened on rollback (no stuck RIT).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-30334: Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS [hbase]
Posted by "nirdosh0110 (via GitHub)" <gi...@apache.org>.
nirdosh0110 commented on PR #8583:
URL: https://github.com/apache/hbase/pull/8583#issuecomment-5684015682
> > What is the order for PR merges we are following for the relevant work @Apache9 @apurtell?
> >
> > * [HBASE-30334: Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS #8583](https://github.com/apache/hbase/pull/8583)
> > * [HBASE-30335 Seed master flushedSequenceIdByRegion with openSeqNum on region OPEN #8584](https://github.com/apache/hbase/pull/8584)
> > * [HBASE-30352: Tolerate stale recovered.edits below durable seqid in split/merge #8599](https://github.com/apache/hbase/pull/8599)
>
@nirdosh0110 what is the sequence of PRs to check-in one by one?
Hi @virajjasani,
- We'd need one more review approval on #8599 before merging it in.
- There are no compile/test dependencies between the three PRs — they touch different files and different subsystems. However, there is a clear safety ordering we should follow. Suggested merge order:
1. #8583 — Rollback fix; prevents stuck RITs.
2. #8599 — Tolerate stale recovered.edits; provides immediate prod relief. Blocker: resolve Apache9's HFile metadata request first.
3. #8584 — Seed durability state on OPEN; addresses the root cause. This is the largest change, so it should be merged last.
Let me know your thoughts. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-30334 Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS [hbase]
Posted by "nirdosh0110 (via GitHub)" <gi...@apache.org>.
nirdosh0110 commented on PR #8583:
URL: https://github.com/apache/hbase/pull/8583#issuecomment-5541414937
^ bumping this up for reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-30334 Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #8583:
URL: https://github.com/apache/hbase/pull/8583#issuecomment-5636912326
What is the order for PR merges we are following for the relevant work @Apache9 @apurtell?
- https://github.com/apache/hbase/pull/8583
- https://github.com/apache/hbase/pull/8584
- https://github.com/apache/hbase/pull/8599
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-30334: Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani merged PR #8583:
URL: https://github.com/apache/hbase/pull/8583
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-30334 Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #8583:
URL: https://github.com/apache/hbase/pull/8583#discussion_r3903378239
##########
hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestMergeTableRegionsProcedure.java:
##########
@@ -327,6 +333,80 @@ public void testRollbackAndDoubleExecution() throws Exception {
assertEquals(initialRegionCount, regions.size());
}
+ /**
+ * HBASE-30334 repro. Plant a stale recovered.edits file on a parent region so that
+ * MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS throws (the exact failure from the 49-min
+ * stuck-RIT incident). After rollback the parents MUST be OPEN. If they stay stuck
+ * (e.g. CLOSED / MERGING), we've reproduced the incident locally and the rollback path
+ * as-is is broken.
+ */
+ @Test
+ public void testRollbackReopensParentsAfterCheckClosedRegionsFailure() throws Exception {
+ final TableName tableName = TableName.valueOf(testMethodName);
+ UTIL.createTable(tableName, new byte[][] { HConstants.CATALOG_FAMILY },
+ new byte[][] { new byte[] { 'b' } });
+ UTIL.waitUntilAllRegionsAssigned(tableName);
+
+ List<RegionInfo> ris = MetaTableAccessor.getTableRegions(UTIL.getConnection(), tableName);
+ assertEquals(2, ris.size());
+ RegionInfo[] regionsToMerge = new RegionInfo[] { ris.get(0), ris.get(1) };
+
+ Configuration conf = UTIL.getConfiguration();
+ Path regionDir =
+ FSUtils.getRegionDirFromRootDir(CommonFSUtils.getRootDir(conf), regionsToMerge[0]);
+ Path recoveredEditsDir = WALSplitUtil.getRegionDirRecoveredEditsDir(regionDir);
+ FileSystem fs = CommonFSUtils.getRootDirFileSystem(conf);
+ fs.mkdirs(recoveredEditsDir);
+ Path staleFile = new Path(recoveredEditsDir, "0000000000000000001");
+ fs.createNewFile(staleFile);
+ assertTrue(WALSplitUtil.hasRecoveredEdits(conf, regionsToMerge[0]),
+ "stale recovered.edits file must be visible");
+
+ AssignmentManager am = UTIL.getHBaseCluster().getMaster().getAssignmentManager();
+ LOG.info("HBASE-30334-DEBUG pre-merge: {} state={} / {} state={}",
+ regionsToMerge[0].getEncodedName(),
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState(),
+ regionsToMerge[1].getEncodedName(),
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState());
+
+ final ProcedureExecutor<MasterProcedureEnv> procExec = getMasterProcedureExecutor();
+ MergeTableRegionsProcedure proc =
+ new MergeTableRegionsProcedure(procExec.getEnvironment(), regionsToMerge, true);
+ long procId = procExec.submitProcedure(proc);
+ ProcedureTestingUtility.waitProcedure(procExec, procId);
+
+ RegionState.State s0Post =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState();
+ RegionState.State s1Post =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState();
+ LOG.info("HBASE-30334-DEBUG post-rollback: {} state={} / {} state={}",
+ regionsToMerge[0].getEncodedName(), s0Post,
+ regionsToMerge[1].getEncodedName(), s1Post);
+
+ ProcedureTestingUtility.assertProcFailed(procExec, procId);
+
+ fs.delete(staleFile, false);
+
+ long elapsed = org.apache.hadoop.hbase.Waiter.waitFor(conf, 30_000, 500, false,
Review Comment:
Just use UTIL.waitFor?
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java:
##########
@@ -284,9 +284,13 @@ protected void rollbackState(final MasterProcedureEnv env, final MergeTableRegio
cleanupMergedRegion(env);
break;
case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS:
+ rollbackCloseRegionsForMerge(env);
break;
case MERGE_TABLE_REGIONS_CLOSE_REGIONS:
- rollbackCloseRegionsForMerge(env);
+ // If it rolls back with state MERGE_TABLE_REGIONS_CLOSE_REGIONS, no need to call
+ // rollbackCloseRegionsForMerge(), otherwise, it will result in duplicate
+ // TransitRegionStateProcedures for parents that are already OPENING. Mirrors
Review Comment:
Already opened?
##########
hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestMergeTableRegionsProcedure.java:
##########
@@ -327,6 +333,80 @@ public void testRollbackAndDoubleExecution() throws Exception {
assertEquals(initialRegionCount, regions.size());
}
+ /**
+ * HBASE-30334 repro. Plant a stale recovered.edits file on a parent region so that
+ * MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS throws (the exact failure from the 49-min
+ * stuck-RIT incident). After rollback the parents MUST be OPEN. If they stay stuck
+ * (e.g. CLOSED / MERGING), we've reproduced the incident locally and the rollback path
+ * as-is is broken.
+ */
+ @Test
+ public void testRollbackReopensParentsAfterCheckClosedRegionsFailure() throws Exception {
+ final TableName tableName = TableName.valueOf(testMethodName);
+ UTIL.createTable(tableName, new byte[][] { HConstants.CATALOG_FAMILY },
+ new byte[][] { new byte[] { 'b' } });
+ UTIL.waitUntilAllRegionsAssigned(tableName);
+
+ List<RegionInfo> ris = MetaTableAccessor.getTableRegions(UTIL.getConnection(), tableName);
+ assertEquals(2, ris.size());
+ RegionInfo[] regionsToMerge = new RegionInfo[] { ris.get(0), ris.get(1) };
+
+ Configuration conf = UTIL.getConfiguration();
+ Path regionDir =
+ FSUtils.getRegionDirFromRootDir(CommonFSUtils.getRootDir(conf), regionsToMerge[0]);
+ Path recoveredEditsDir = WALSplitUtil.getRegionDirRecoveredEditsDir(regionDir);
+ FileSystem fs = CommonFSUtils.getRootDirFileSystem(conf);
+ fs.mkdirs(recoveredEditsDir);
+ Path staleFile = new Path(recoveredEditsDir, "0000000000000000001");
+ fs.createNewFile(staleFile);
+ assertTrue(WALSplitUtil.hasRecoveredEdits(conf, regionsToMerge[0]),
+ "stale recovered.edits file must be visible");
+
+ AssignmentManager am = UTIL.getHBaseCluster().getMaster().getAssignmentManager();
+ LOG.info("HBASE-30334-DEBUG pre-merge: {} state={} / {} state={}",
+ regionsToMerge[0].getEncodedName(),
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState(),
+ regionsToMerge[1].getEncodedName(),
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState());
+
+ final ProcedureExecutor<MasterProcedureEnv> procExec = getMasterProcedureExecutor();
+ MergeTableRegionsProcedure proc =
+ new MergeTableRegionsProcedure(procExec.getEnvironment(), regionsToMerge, true);
+ long procId = procExec.submitProcedure(proc);
+ ProcedureTestingUtility.waitProcedure(procExec, procId);
+
+ RegionState.State s0Post =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState();
+ RegionState.State s1Post =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState();
+ LOG.info("HBASE-30334-DEBUG post-rollback: {} state={} / {} state={}",
+ regionsToMerge[0].getEncodedName(), s0Post,
+ regionsToMerge[1].getEncodedName(), s1Post);
+
+ ProcedureTestingUtility.assertProcFailed(procExec, procId);
+
+ fs.delete(staleFile, false);
+
+ long elapsed = org.apache.hadoop.hbase.Waiter.waitFor(conf, 30_000, 500, false,
+ new org.apache.hadoop.hbase.Waiter.Predicate<Exception>() {
+ @Override
+ public boolean evaluate() {
+ RegionState.State a =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState();
+ RegionState.State b =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState();
+ return a == RegionState.State.OPEN && b == RegionState.State.OPEN;
+ }
+ });
+
+ assertTrue(elapsed > 0,
Review Comment:
What is the intention for asserting elapsed > 0?
##########
hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestMergeTableRegionsProcedure.java:
##########
@@ -327,6 +333,80 @@ public void testRollbackAndDoubleExecution() throws Exception {
assertEquals(initialRegionCount, regions.size());
}
+ /**
+ * HBASE-30334 repro. Plant a stale recovered.edits file on a parent region so that
+ * MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS throws (the exact failure from the 49-min
+ * stuck-RIT incident). After rollback the parents MUST be OPEN. If they stay stuck
+ * (e.g. CLOSED / MERGING), we've reproduced the incident locally and the rollback path
+ * as-is is broken.
+ */
+ @Test
+ public void testRollbackReopensParentsAfterCheckClosedRegionsFailure() throws Exception {
+ final TableName tableName = TableName.valueOf(testMethodName);
+ UTIL.createTable(tableName, new byte[][] { HConstants.CATALOG_FAMILY },
+ new byte[][] { new byte[] { 'b' } });
+ UTIL.waitUntilAllRegionsAssigned(tableName);
+
+ List<RegionInfo> ris = MetaTableAccessor.getTableRegions(UTIL.getConnection(), tableName);
+ assertEquals(2, ris.size());
+ RegionInfo[] regionsToMerge = new RegionInfo[] { ris.get(0), ris.get(1) };
+
+ Configuration conf = UTIL.getConfiguration();
+ Path regionDir =
+ FSUtils.getRegionDirFromRootDir(CommonFSUtils.getRootDir(conf), regionsToMerge[0]);
+ Path recoveredEditsDir = WALSplitUtil.getRegionDirRecoveredEditsDir(regionDir);
+ FileSystem fs = CommonFSUtils.getRootDirFileSystem(conf);
+ fs.mkdirs(recoveredEditsDir);
+ Path staleFile = new Path(recoveredEditsDir, "0000000000000000001");
+ fs.createNewFile(staleFile);
+ assertTrue(WALSplitUtil.hasRecoveredEdits(conf, regionsToMerge[0]),
+ "stale recovered.edits file must be visible");
+
+ AssignmentManager am = UTIL.getHBaseCluster().getMaster().getAssignmentManager();
+ LOG.info("HBASE-30334-DEBUG pre-merge: {} state={} / {} state={}",
+ regionsToMerge[0].getEncodedName(),
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState(),
+ regionsToMerge[1].getEncodedName(),
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState());
+
+ final ProcedureExecutor<MasterProcedureEnv> procExec = getMasterProcedureExecutor();
+ MergeTableRegionsProcedure proc =
+ new MergeTableRegionsProcedure(procExec.getEnvironment(), regionsToMerge, true);
+ long procId = procExec.submitProcedure(proc);
+ ProcedureTestingUtility.waitProcedure(procExec, procId);
+
+ RegionState.State s0Post =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState();
+ RegionState.State s1Post =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState();
+ LOG.info("HBASE-30334-DEBUG post-rollback: {} state={} / {} state={}",
+ regionsToMerge[0].getEncodedName(), s0Post,
+ regionsToMerge[1].getEncodedName(), s1Post);
+
+ ProcedureTestingUtility.assertProcFailed(procExec, procId);
+
+ fs.delete(staleFile, false);
+
+ long elapsed = org.apache.hadoop.hbase.Waiter.waitFor(conf, 30_000, 500, false,
+ new org.apache.hadoop.hbase.Waiter.Predicate<Exception>() {
Review Comment:
Use lambda?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-30334 Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS [hbase]
Posted by "nirdosh0110 (via GitHub)" <gi...@apache.org>.
nirdosh0110 commented on code in PR #8583:
URL: https://github.com/apache/hbase/pull/8583#discussion_r3909786065
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java:
##########
@@ -284,7 +284,6 @@ protected void rollbackState(final MasterProcedureEnv env, final MergeTableRegio
cleanupMergedRegion(env);
break;
case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS:
- break;
Review Comment:
Not applicable now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-30334 Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS [hbase]
Posted by "nirdosh0110 (via GitHub)" <gi...@apache.org>.
nirdosh0110 commented on code in PR #8583:
URL: https://github.com/apache/hbase/pull/8583#discussion_r3905294501
##########
hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestMergeTableRegionsProcedure.java:
##########
@@ -327,6 +333,80 @@ public void testRollbackAndDoubleExecution() throws Exception {
assertEquals(initialRegionCount, regions.size());
}
+ /**
+ * HBASE-30334 repro. Plant a stale recovered.edits file on a parent region so that
+ * MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS throws (the exact failure from the 49-min
+ * stuck-RIT incident). After rollback the parents MUST be OPEN. If they stay stuck
+ * (e.g. CLOSED / MERGING), we've reproduced the incident locally and the rollback path
+ * as-is is broken.
+ */
+ @Test
+ public void testRollbackReopensParentsAfterCheckClosedRegionsFailure() throws Exception {
+ final TableName tableName = TableName.valueOf(testMethodName);
+ UTIL.createTable(tableName, new byte[][] { HConstants.CATALOG_FAMILY },
+ new byte[][] { new byte[] { 'b' } });
+ UTIL.waitUntilAllRegionsAssigned(tableName);
+
+ List<RegionInfo> ris = MetaTableAccessor.getTableRegions(UTIL.getConnection(), tableName);
+ assertEquals(2, ris.size());
+ RegionInfo[] regionsToMerge = new RegionInfo[] { ris.get(0), ris.get(1) };
+
+ Configuration conf = UTIL.getConfiguration();
+ Path regionDir =
+ FSUtils.getRegionDirFromRootDir(CommonFSUtils.getRootDir(conf), regionsToMerge[0]);
+ Path recoveredEditsDir = WALSplitUtil.getRegionDirRecoveredEditsDir(regionDir);
+ FileSystem fs = CommonFSUtils.getRootDirFileSystem(conf);
+ fs.mkdirs(recoveredEditsDir);
+ Path staleFile = new Path(recoveredEditsDir, "0000000000000000001");
+ fs.createNewFile(staleFile);
+ assertTrue(WALSplitUtil.hasRecoveredEdits(conf, regionsToMerge[0]),
+ "stale recovered.edits file must be visible");
+
+ AssignmentManager am = UTIL.getHBaseCluster().getMaster().getAssignmentManager();
+ LOG.info("HBASE-30334-DEBUG pre-merge: {} state={} / {} state={}",
+ regionsToMerge[0].getEncodedName(),
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState(),
+ regionsToMerge[1].getEncodedName(),
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState());
+
+ final ProcedureExecutor<MasterProcedureEnv> procExec = getMasterProcedureExecutor();
+ MergeTableRegionsProcedure proc =
+ new MergeTableRegionsProcedure(procExec.getEnvironment(), regionsToMerge, true);
+ long procId = procExec.submitProcedure(proc);
+ ProcedureTestingUtility.waitProcedure(procExec, procId);
+
+ RegionState.State s0Post =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState();
+ RegionState.State s1Post =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState();
+ LOG.info("HBASE-30334-DEBUG post-rollback: {} state={} / {} state={}",
+ regionsToMerge[0].getEncodedName(), s0Post,
+ regionsToMerge[1].getEncodedName(), s1Post);
+
+ ProcedureTestingUtility.assertProcFailed(procExec, procId);
+
+ fs.delete(staleFile, false);
+
+ long elapsed = org.apache.hadoop.hbase.Waiter.waitFor(conf, 30_000, 500, false,
+ new org.apache.hadoop.hbase.Waiter.Predicate<Exception>() {
+ @Override
+ public boolean evaluate() {
+ RegionState.State a =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState();
+ RegionState.State b =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState();
+ return a == RegionState.State.OPEN && b == RegionState.State.OPEN;
+ }
+ });
+
+ assertTrue(elapsed > 0,
Review Comment:
Good catch — I've dropped this change from the test case. It looks like it was leftover from local testing and got pushed inadvertently. Thanks for flagging!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-30334 Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS [hbase]
Posted by "nirdosh0110 (via GitHub)" <gi...@apache.org>.
nirdosh0110 commented on code in PR #8583:
URL: https://github.com/apache/hbase/pull/8583#discussion_r3905310464
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java:
##########
@@ -284,9 +284,13 @@ protected void rollbackState(final MasterProcedureEnv env, final MergeTableRegio
cleanupMergedRegion(env);
break;
case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS:
+ rollbackCloseRegionsForMerge(env);
break;
case MERGE_TABLE_REGIONS_CLOSE_REGIONS:
- rollbackCloseRegionsForMerge(env);
+ // If it rolls back with state MERGE_TABLE_REGIONS_CLOSE_REGIONS, no need to call
+ // rollbackCloseRegionsForMerge(), otherwise, it will result in duplicate
+ // TransitRegionStateProcedures for parents that are already OPENING. Mirrors
Review Comment:
Thanks for catching. Updated the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-30334 Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #8583:
URL: https://github.com/apache/hbase/pull/8583#discussion_r3899987410
##########
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java:
##########
@@ -284,7 +284,6 @@ protected void rollbackState(final MasterProcedureEnv env, final MergeTableRegio
cleanupMergedRegion(env);
break;
case MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS:
- break;
Review Comment:
Why we need this change? In general, when rolling back a procedure, finally we will arrive the MERGE_TABLE_REGIONS_CLOSE_REGIONS state and reopen the parent regions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-30334 Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS [hbase]
Posted by "nirdosh0110 (via GitHub)" <gi...@apache.org>.
nirdosh0110 commented on code in PR #8583:
URL: https://github.com/apache/hbase/pull/8583#discussion_r3905300973
##########
hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestMergeTableRegionsProcedure.java:
##########
@@ -327,6 +333,80 @@ public void testRollbackAndDoubleExecution() throws Exception {
assertEquals(initialRegionCount, regions.size());
}
+ /**
+ * HBASE-30334 repro. Plant a stale recovered.edits file on a parent region so that
+ * MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS throws (the exact failure from the 49-min
+ * stuck-RIT incident). After rollback the parents MUST be OPEN. If they stay stuck
+ * (e.g. CLOSED / MERGING), we've reproduced the incident locally and the rollback path
+ * as-is is broken.
+ */
+ @Test
+ public void testRollbackReopensParentsAfterCheckClosedRegionsFailure() throws Exception {
+ final TableName tableName = TableName.valueOf(testMethodName);
+ UTIL.createTable(tableName, new byte[][] { HConstants.CATALOG_FAMILY },
+ new byte[][] { new byte[] { 'b' } });
+ UTIL.waitUntilAllRegionsAssigned(tableName);
+
+ List<RegionInfo> ris = MetaTableAccessor.getTableRegions(UTIL.getConnection(), tableName);
+ assertEquals(2, ris.size());
+ RegionInfo[] regionsToMerge = new RegionInfo[] { ris.get(0), ris.get(1) };
+
+ Configuration conf = UTIL.getConfiguration();
+ Path regionDir =
+ FSUtils.getRegionDirFromRootDir(CommonFSUtils.getRootDir(conf), regionsToMerge[0]);
+ Path recoveredEditsDir = WALSplitUtil.getRegionDirRecoveredEditsDir(regionDir);
+ FileSystem fs = CommonFSUtils.getRootDirFileSystem(conf);
+ fs.mkdirs(recoveredEditsDir);
+ Path staleFile = new Path(recoveredEditsDir, "0000000000000000001");
+ fs.createNewFile(staleFile);
+ assertTrue(WALSplitUtil.hasRecoveredEdits(conf, regionsToMerge[0]),
+ "stale recovered.edits file must be visible");
+
+ AssignmentManager am = UTIL.getHBaseCluster().getMaster().getAssignmentManager();
+ LOG.info("HBASE-30334-DEBUG pre-merge: {} state={} / {} state={}",
+ regionsToMerge[0].getEncodedName(),
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState(),
+ regionsToMerge[1].getEncodedName(),
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState());
+
+ final ProcedureExecutor<MasterProcedureEnv> procExec = getMasterProcedureExecutor();
+ MergeTableRegionsProcedure proc =
+ new MergeTableRegionsProcedure(procExec.getEnvironment(), regionsToMerge, true);
+ long procId = procExec.submitProcedure(proc);
+ ProcedureTestingUtility.waitProcedure(procExec, procId);
+
+ RegionState.State s0Post =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState();
+ RegionState.State s1Post =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState();
+ LOG.info("HBASE-30334-DEBUG post-rollback: {} state={} / {} state={}",
+ regionsToMerge[0].getEncodedName(), s0Post,
+ regionsToMerge[1].getEncodedName(), s1Post);
+
+ ProcedureTestingUtility.assertProcFailed(procExec, procId);
+
+ fs.delete(staleFile, false);
+
+ long elapsed = org.apache.hadoop.hbase.Waiter.waitFor(conf, 30_000, 500, false,
+ new org.apache.hadoop.hbase.Waiter.Predicate<Exception>() {
Review Comment:
Addressed in the latest commit — please take another look when you get a chance. Thanks!
##########
hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestMergeTableRegionsProcedure.java:
##########
@@ -327,6 +333,80 @@ public void testRollbackAndDoubleExecution() throws Exception {
assertEquals(initialRegionCount, regions.size());
}
+ /**
+ * HBASE-30334 repro. Plant a stale recovered.edits file on a parent region so that
+ * MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS throws (the exact failure from the 49-min
+ * stuck-RIT incident). After rollback the parents MUST be OPEN. If they stay stuck
+ * (e.g. CLOSED / MERGING), we've reproduced the incident locally and the rollback path
+ * as-is is broken.
+ */
+ @Test
+ public void testRollbackReopensParentsAfterCheckClosedRegionsFailure() throws Exception {
+ final TableName tableName = TableName.valueOf(testMethodName);
+ UTIL.createTable(tableName, new byte[][] { HConstants.CATALOG_FAMILY },
+ new byte[][] { new byte[] { 'b' } });
+ UTIL.waitUntilAllRegionsAssigned(tableName);
+
+ List<RegionInfo> ris = MetaTableAccessor.getTableRegions(UTIL.getConnection(), tableName);
+ assertEquals(2, ris.size());
+ RegionInfo[] regionsToMerge = new RegionInfo[] { ris.get(0), ris.get(1) };
+
+ Configuration conf = UTIL.getConfiguration();
+ Path regionDir =
+ FSUtils.getRegionDirFromRootDir(CommonFSUtils.getRootDir(conf), regionsToMerge[0]);
+ Path recoveredEditsDir = WALSplitUtil.getRegionDirRecoveredEditsDir(regionDir);
+ FileSystem fs = CommonFSUtils.getRootDirFileSystem(conf);
+ fs.mkdirs(recoveredEditsDir);
+ Path staleFile = new Path(recoveredEditsDir, "0000000000000000001");
+ fs.createNewFile(staleFile);
+ assertTrue(WALSplitUtil.hasRecoveredEdits(conf, regionsToMerge[0]),
+ "stale recovered.edits file must be visible");
+
+ AssignmentManager am = UTIL.getHBaseCluster().getMaster().getAssignmentManager();
+ LOG.info("HBASE-30334-DEBUG pre-merge: {} state={} / {} state={}",
+ regionsToMerge[0].getEncodedName(),
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState(),
+ regionsToMerge[1].getEncodedName(),
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState());
+
+ final ProcedureExecutor<MasterProcedureEnv> procExec = getMasterProcedureExecutor();
+ MergeTableRegionsProcedure proc =
+ new MergeTableRegionsProcedure(procExec.getEnvironment(), regionsToMerge, true);
+ long procId = procExec.submitProcedure(proc);
+ ProcedureTestingUtility.waitProcedure(procExec, procId);
+
+ RegionState.State s0Post =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[0]).getState();
+ RegionState.State s1Post =
+ am.getRegionStates().getRegionStateNode(regionsToMerge[1]).getState();
+ LOG.info("HBASE-30334-DEBUG post-rollback: {} state={} / {} state={}",
+ regionsToMerge[0].getEncodedName(), s0Post,
+ regionsToMerge[1].getEncodedName(), s1Post);
+
+ ProcedureTestingUtility.assertProcFailed(procExec, procId);
+
+ fs.delete(staleFile, false);
+
+ long elapsed = org.apache.hadoop.hbase.Waiter.waitFor(conf, 30_000, 500, false,
Review Comment:
Addressed in the latest commit — please take another look when you get a chance. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-30334: Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONS [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on PR #8583:
URL: https://github.com/apache/hbase/pull/8583#issuecomment-5683813049
> What is the order for PR merges we are following for the relevant work @Apache9 @apurtell?
>
> * [HBASE-30334: Reopen parent regions on rollback of MERGE_TABLE_REGIONS_CHECK_CLOSED_REGIONSÂ #8583](https://github.com/apache/hbase/pull/8583)
> * [HBASE-30335 Seed master flushedSequenceIdByRegion with openSeqNum on region OPENÂ #8584](https://github.com/apache/hbase/pull/8584)
> * [HBASE-30352: Tolerate stale recovered.edits below durable seqid in split/merge #8599](https://github.com/apache/hbase/pull/8599)
@Apache9 we have run into this issue in prod again. Let's move forward with these PRs?
@nirdosh0110 what is the sequence of PRs to check-in one by one? I can take it forward accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org