[jira] [Created] (HDDS-16929) SCM follower stops applying the Raft log, while reporting RUNNING, if the transaction buffer init fails after a checkpoint install

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16929:
---------------------------------

             Summary: SCM follower stops applying the Raft log, while reporting RUNNING, if the transaction buffer init fails after a checkpoint install
                 Key: HDDS-16929
                 URL: https://issues.apache.org/jira/browse/HDDS-16929
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-5-fail-reinitialize-when-transaction-buffer-init-fails.patch, TestBugMC5BufferInitFailureStaleApplied.java

h3. Mechanism
{{SCMStateMachine.reinitialize}} installs the downloaded checkpoint and then calls {{transactionBuffer.init()}} and {{setLastAppliedTermIndex}} inside a try block whose catch only logs "Failed to unpause". If {{init()}} throws, the state machine still resumes with the transaction info and latest snapshot from before the install. Ratis {{StateMachineUpdater.reload}} then takes that old snapshot index (-1 for a new SCM) as the applied index. Ratis purged the follower's log up to the checkpoint index when it acknowledged the install, so the entry after the applied index no longer exists and the updater never applies anything again. The leader does not send another install notification, because the follower's log is consistent with its own.

h3. Trigger
# A follower SCM installs the leader's checkpoint through the normal Ratis install notification (for example a new SCM bootstrapping, or a lagging follower with {{ozone.scm.ha.ratis.log.purge.enabled}}).
# {{SCMHADBTransactionBufferImpl.init}} fails during {{reinitialize}}, for example a read error on the transaction info key of the newly installed DB.

The second step is unlikely: {{installCheckpoint}} reads and decodes the same key from the checkpoint moments earlier, so it needs a transient read error at that instant. In the reproduction it is injected.

h3. Impact
* The follower reports RUNNING but never applies another entry and never starts its datanode server. It is missing every container created after the install.
* A graceful stop of that SCM never completes, because {{StateMachineUpdater}} waits to reach the commit index; the updater thread also spins on one core. It has to be killed. Argued, not run: after a restart it opens the installed DB and catches up, unless it was elected leader in the meantime, because the flush on becoming leader ({{SCMStateMachine.notifyLeaderChanged}}) writes the stale transaction info into that DB.
* Argued, not run for this trigger: its log is complete, so it can be elected leader, and then no SCM is a ready leader. I ran that outcome for the same applied index below log start state reached through a failed checkpoint copy.

h3. Reproduction
PASS on unmodified source. [^TestBugMC5BufferInitFailureStaleApplied.java] ({{hadoop-ozone/integration-test}}) uses the cluster shape of {{TestSCMInstallSnapshotWithHA}} (three SCMs, one started late, so it installs the leader's checkpoint through Ratis; log purge enabled, purge gap 5, snapshot threshold 5) and injects one {{RocksDatabaseException}} into the transaction info read that {{SCMHADBTransactionBufferImpl.init}} does inside {{reinitialize}}, with a Mockito spy on the follower's {{SCMMetadataStore}}. Result: lifecycle RUNNING, DB transaction index 202, Ratis applied index -1, log start 203, commit 238, 16 of 16 later containers missing, datanode server not started, about 4.9 s of updater CPU in 5 s, graceful stop still running after 20 s. A passing test means the defect is present.

h3. Patch
[^MC-5-fail-reinitialize-when-transaction-buffer-init-fails.patch], against b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at 52812559dad. It conflicts textually with the patch attached to HDDS-16923 in {{SCMStateMachine}} and {{TestSCMStateMachine}}; whichever lands second needs a rebase.

The patch removes the catch, so the failure propagates from {{reinitialize}} like a failure of {{installCheckpoint}} already does. Ratis then closes the server and SCM shuts down; on restart it opens the installed DB, whose transaction info is the checkpoint index, and catches up. With the patch the reproduction fails at its first assertion because the follower SCM stops, and a restart of that follower applied up to the leader's index and started its datanode server (local variant of the reproduction, not attached).

Covered by {{testReinitializeFailsWhenTransactionBufferInitFails}} and {{testReinitializeResumesAtInstalledCheckpoint}} in the existing {{TestSCMStateMachine}}. Without the change the first fails with "Expected java.io.IOException to be thrown, but nothing was thrown." With it {{TestSCMStateMachine}}, {{TestSCMHAManagerImpl}} and {{TestSCMHADBTransactionBufferImpl}} (31 tests) and the mini cluster suites {{TestSCMInstallSnapshotWithHA}} and {{TestSCMInstallSnapshot}} (5 tests) pass, checkstyle is clean.

Found by TLA+ model checking and code review of the SCM Ratis snapshot and checkpoint install paths under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS and RATIS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org