You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/10/12 04:50:00 UTC

[jira] [Created] (HDDS-16928) SCM never catches up after a checkpoint install that fails or is interrupted once Ratis has acknowledged it, even after a restart

Siyao Meng created HDDS-16928:
---------------------------------

             Summary: SCM never catches up after a checkpoint install that fails or is interrupted once Ratis has acknowledged it, even after a restart
                 Key: HDDS-16928
                 URL: https://issues.apache.org/jira/browse/HDDS-16928
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-1-test-follower-scm-catches-up-after-failed-checkpoint-install.patch, TestBugMC1FailedInstallWedgesFollower.java

h3. Mechanism
{{SCMStateMachine.notifyInstallSnapshotFromLeader}} completes its future with the checkpoint TermIndex as soon as the leader's checkpoint is downloaded and verified. The DB swap is deferred to {{SCMStateMachine.reinitialize}}, which Ratis runs later on the {{StateMachineUpdater}} thread. On the next install notification Ratis (3.3.1) pauses the state machine and signals the {{StateMachineUpdater}} thread to run {{reinitialize}}, but does not wait for it: in the same call it purges the follower's Raft log up to the checkpoint index and replies SNAPSHOT_INSTALLED, and the leader continues with the entries after that index. {{reinitialize}} then calls {{SCMHAManagerImpl.installCheckpoint}}.

If that install fails, every failure path keeps the old DB (failed backup move, failed copy with restore, failed reload with rollback), and SCM shuts down. The same holds if the process is killed after the acknowledgement and before {{HAUtils.replaceDBWithCheckpoint}} moves the old DB aside (argued, not run); a kill during the copy leaves the {{dbInconsistentMarker}} file, and the SCM then refuses to start. The pending checkpoint was only held in memory. After a restart the applied index is the old DB's (for a new SCM, -1) while the Raft log starts after the checkpoint index, so {{StateMachineUpdater}} never finds the next entry to apply. The leader does not send another install notification, because the follower's next index is already inside the leader's log.

h3. Trigger
# An SCM installs the leader's checkpoint through the Ratis install notification. At default settings ({{ozone.scm.ha.ratis.log.purge.enabled=false}}) this is in practice an SCM that is bootstrapping (new or replaced node); with purge enabled, also a follower that fell behind.
# The leader commits at least one more entry that reaches this SCM before the install completes (any SCM write traffic).
# {{installCheckpoint}} fails with the old DB kept, or the process is killed after the acknowledgement and before the old DB is moved aside (a kill during the copy leaves the {{dbInconsistentMarker}}, and the SCM then refuses to start). Realistic causes are a full disk during the copy of the whole DB, an I/O error, or a failure while reloading the SCM services from the new DB. In the reproduction the failure is forced with a leftover {{dbInconsistentMarker}} file in the DB directory, which makes {{HAUtils.replaceDBWithCheckpoint}} fail and restore the old DB. The kill case was not run. For a bootstrapping SCM this needs the leader to have added it to the Raft group, which happens within moments of the acknowledgement, long before a copy of a real DB ends; when the injected immediate failure came first, the bootstrap did not complete instead (not followed further).
# The SCM is restarted.

h3. Impact
* The restarted SCM never applies another transaction and never starts its datanode server, with no error beyond the periodic "Deferred datanode-server start is still waiting for follower catch-up" warning. Nothing recovers it automatically.
* A graceful stop of that SCM never completes ({{StateMachineUpdater}} waits to reach the commit index), and its updater thread spins on one core.
* Its Raft log is complete, so it can become leader. After a leadership transfer to it (admin {{transferLeadership}}), it was the Ratis leader and no SCM reported itself as a ready leader for the 20 s observed, so the SCM service was unavailable. I did not run an election that picks it without the transfer.
* Recovery needs an operator: kill it, remove its SCM DB and Ratis storage, and bootstrap it again (argued, not run).

h3. Reproduction
PASS on unmodified source, default configuration. [^TestBugMC1FailedInstallWedgesFollower.java] ({{hadoop-ozone/integration-test}}) starts three SCMs with one inactive, places the marker file in the inactive SCM's DB directory, starts it while the leader keeps allocating containers, waits until it shuts down after the failed install, removes the file and restarts it. Result: SNAPSHOT_INSTALLED and the leader moving to the next index logged before the install ran; after the restart applied index -1 before and after, log start 90, commit 192 against leader applied 191, 0 install notifications, datanode server not started, 0 of 80 containers, about 9.7 s of updater CPU in 10 s; after a leadership transfer to it, it was the Ratis leader and no SCM was a ready leader; a graceful stop still running after 20 s. In this JVM the restarted SCM gets new datanode and gRPC ports, because the stopped instance keeps them bound; a process exit releases them. If the injected failure comes before the leader has added the SCM to the Raft group, the test aborts and has to be rerun. A passing test means the defect is present.

h3. Suggested fix
SCM cannot undo the purge and the acknowledgement from {{reinitialize}}. Recommended: install the checkpoint inside the notification future, as {{OzoneManagerStateMachine.notifyInstallSnapshotFromLeader}} does, and fail the future on any error, so Ratis neither purges nor acknowledges a failed install. This reverses the deferral chosen in HDDS-5173 and needs the updater to be quiesced during the swap. It should not copy OM's handling of a failed install, which completes the future with null (HDDS-11470). As a safety net, SCM should check at startup whether its applied index is below the start of its Raft log and fail with a clear message; that also catches the same state reached through other paths, such as a transaction buffer init failure in {{reinitialize}} or a reload failure that rolls back to the old DB.

Alternatives: change Ratis to purge and reply SNAPSHOT_INSTALLED only after {{reinitialize}} has succeeded, or persist the pending checkpoint before acknowledging and install it at startup.

h3. Patch
[^MC-1-test-follower-scm-catches-up-after-failed-checkpoint-install.patch], against b0aa6475b7856d2c7f467dd3db12b9a1dd116f63, test only. It also applies to master at 52812559dad.

It adds {{testSCMCatchesUpAfterFailedCheckpointInstall}} to {{TestStorageContainerManagerHA}}. A follower SCM that is a voting member falls behind the start of the leader's log (log purge enabled, one snapshot requested on the leader), and when it restarts its install fails once in {{installCheckpoint}} with the old DB in place (its metadata store, set through {{SCMConfigurator.setMetadataStore}}, refuses to stop once). After another restart the follower must catch up with the leader. On the current code it fails with "Timed out waiting for condition": the follower stays at applied index 42 while the leader is at 1475 and the follower's log starts at 1261, and the teardown then takes several minutes (about 5) because the wedged SCM's stop does not complete. It is meant to land with the fix. A member follower is used instead of the bootstrap case of the reproduction because an injected failure is immediate, and for a bootstrapping SCM it can come before the leader has added the SCM to the Raft group.

Found by TLA+ model checking and code review of the SCM Ratis snapshot and checkpoint install paths under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS and RATIS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a test proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org