[jira] [Updated] (HDDS-16927) SCM never deletes the checkpoint it downloads for a Ratis snapshot install, leaving a full DB copy in its snapshot directory per install

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16927:
------------------------------
    Attachment: MC-8-delete-downloaded-scm-checkpoint-after-install.patch

> SCM never deletes the checkpoint it downloads for a Ratis snapshot install, leaving a full DB copy in its snapshot directory per install
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16927
>                 URL: https://issues.apache.org/jira/browse/HDDS-16927
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Minor
>         Attachments: MC-8-delete-downloaded-scm-checkpoint-after-install.patch, TestBugMC8CheckpointDirLeak.java
>
>
> h3. Mechanism
> {{SCMSnapshotProvider.getSCMDBSnapshot}} downloads the leader's checkpoint into the SCM Ratis snapshot directory ({{ozone.scm.ha.ratis.snapshot.dir}}), untars it to {{scm.db-<leader>-<time>}} and deletes only the tarball. {{HAUtils.replaceDBWithCheckpoint}} copies that directory into {{scm.db}} with {{copyDirectory}} and documents that the checkpoint is not deleted there. Its caller {{SCMHAManagerImpl.installCheckpoint}} deletes only the backup of the old DB, and {{SCMStateMachine}} deletes nothing. The failure paths do not delete it either: a failed secret key fetch or a {{verifyCheckpointFromLeader}} that catches an exception and returns null in {{notifyInstallSnapshotFromLeader}}, and an install that is rejected as too old or fails in {{installCheckpoint}}. Only the index check in {{HAUtils.verifyTransactionInfo}} deletes a downloaded checkpoint. SCM startup creates the snapshot directory but never cleans it.
> h3. Trigger
> Any checkpoint install on an SCM. With default settings this happens on every SCM added with {{--bootstrap}}: Ratis (3.3.1) sends a snapshot notification to a bootstrapping follower once. With {{ozone.scm.ha.ratis.log.purge.enabled}} it also happens each time a follower falls behind the purged log.
> h3. Impact
> * Each install leaves one full copy of the SCM DB on the metadata volume, and nothing reclaims it (run for one install per SCM; repeated installs on the same SCM follow from the same code, not run).
> * Repeated installs need free space for the next untar and copy on the same volume, so the leak can eventually make an install fail for lack of space (argued, not run).
> * No effect on correctness.
> h3. Reproduction
> PASS on the unmodified source, default configuration. [^TestBugMC8CheckpointDirLeak.java] ({{hadoop-ozone/integration-test}}) starts three SCMs with one held back and starts it, so it bootstraps and installs the leader's checkpoint. Afterwards its snapshot directory holds one {{scm.db-<leader>-<time>}} directory (70,399 bytes in the run), and the follower that the mini cluster bootstraps at startup holds one too. A passing test means the defect is present. With the patch all snapshot directories are empty after the install and the test fails at its {{hasSize(1)}} assertion.
> h3. Patch
> [^MC-8-delete-downloaded-scm-checkpoint-after-install.patch], against b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at 52812559dad.
> {{SCMStateMachine}} deletes the downloaded checkpoint with {{DBCheckpoint.cleanupCheckpoint()}} once it is not needed any more: in {{reinitialize}} after {{installCheckpoint}} returns or throws, and in {{notifyInstallSnapshotFromLeader}} when the secret key fetch fails or the verification returns null. A failed delete is logged and does not affect the install. This matches the cleanup the OM does in {{OzoneManager.installSnapshotFromLeader}}.
> Not covered by the patch: the tar and partial directory left by a failed untar, the DB copies left by a rollback after a failed reload (SCM exits there), a checkpoint stored by {{setInstallingSnapshotData}} for which Ratis never calls {{reinitialize}} before the SCM stops, for example when the leader changes before its next install notification (code reading), and directories left by earlier versions or crashes, which can be deleted by hand while no install is running.
> Covered by assertions added to {{TestSCMInstallSnapshot.testInstallCheckPoint}}: the checkpoint directory is gone after the in process install, and a checkpoint older than the applied index is rejected and deleted as well. Without the change the first assertion fails because the directory still exists. With it {{TestSCMInstallSnapshot}}, {{TestSCMInstallSnapshotWithHA}}, {{TestSCMSnapshot}}, {{TestSecretKeySnapshot}}, {{TestScmHAFinalization}} and the server-scm {{TestSCMStateMachine}} pass, and checkstyle is clean. This patch and HDDS-16924 both append to {{TestSCMInstallSnapshot.testInstallCheckPoint}}, so whichever lands second needs a trivial rebase.
> Found by TLA+ model checking and code review of the SCM Ratis snapshot and checkpoint install paths under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org