[jira] [Created] (HBASE-30344) [Flaky Test] TestSyncReplicationStandbyKillMaster may verify before RegionServer sync state converges

["mazhengxuan (Jira)" <ji...@apache.org>]

mazhengxuan created HBASE-30344:
-----------------------------------

             Summary: [Flaky Test] TestSyncReplicationStandbyKillMaster may verify before RegionServer sync state converges
                 Key: HBASE-30344
                 URL: https://issues.apache.org/jira/browse/HBASE-30344
             Project: HBase
          Issue Type: Bug
          Components: Replication, test
         Environment: Apache HBase master

GitHub Actions:
JDK17 Hadoop3 Unit Check, large-wave-3

Failed workflow job:
https://github.com/apache/hbase/actions/runs/33056801754/job/98468827602
            Reporter: mazhengxuan
            Assignee: mazhengxuan


TestSyncReplicationStandbyKillMaster is still flaky on master after HBASE-30249.

It failed in the JDK17 Hadoop3 Unit Check for apache/hbase#8483:

https://github.com/apache/hbase/actions/runs/33056801754/job/98468827602

The first run failed in TestSyncReplicationStandbyKillMaster.testStandbyKillMaster. The test waits for the peer state reported by Admin to become DOWNGRADE_ACTIVE and then immediately verifies the replicated data:

await().atMost(Duration.ofMinutes(3))
  .untilAsserted(() -> assertEquals(
    SyncReplicationState.DOWNGRADE_ACTIVE,
    UTIL2.getAdmin().getReplicationPeerSyncReplicationState(PEER_ID)));

verify(UTIL2, 0, COUNT);

The verification request was rejected by the RegionServer because its local sync replication state was still STANDBY:

org.apache.hadoop.hbase.DoNotRetryIOException:
SyncRep,... is in STANDBY state.
  at org.apache.hadoop.hbase.regionserver.RSRpcServices.rejectIfInStandByState(...)
  at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(...)

The Admin result reflects the peer state on the Master side. The RegionServer learns about this state asynchronously, so observing DOWNGRADE_ACTIVE through Admin does not mean that every RegionServer has already applied the transition.

The Surefire retries then failed during setup with:

1. java.io.IOException: Shutting down
2. java.lang.IllegalStateException: A mini-cluster is already running

These look like follow-up failures caused by the cluster cleanup after the first run failed.

HBASE-30249 made the Master failure injection deterministic, but the test still does not wait for the RegionServer-side state transition before calling verify().

The test should wait, with a timeout, until the RegionServer serving the table no longer rejects requests because of the STANDBY state. This should be handled as test synchronization unless further investigation shows that the RegionServer state never converges.

The failure can be reproduced with Surefire retries disabled:

mvn -pl hbase-server -am \
  -Dtest=TestSyncReplicationStandbyKillMaster \
  -Dsurefire.failIfNoSpecifiedTests=false \
  -Dsurefire.rerunFailingTestsCount=0 \
  -DskipITs \
  test

Related:

* HBASE-30249
* https://github.com/apache/hbase/pull/8393
* https://github.com/apache/hbase/pull/8483



--
This message was sent by Atlassian Jira
(v8.20.10#820010)