[jira] [Created] (HDDS-17244) Quota repair skips deleted snapshots that are not purged yet, so the pending deletes they still hold drop out of the bucket's snapshotUsedBytes

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-17244:
---------------------------------

             Summary: Quota repair skips deleted snapshots that are not purged yet, so the pending deletes they still hold drop out of the bucket's snapshotUsedBytes
                 Key: HDDS-17244
                 URL: https://issues.apache.org/jira/browse/HDDS-17244
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: CR-14-quota-repair-scans-deleted-snapshots-not-yet-purged.patch, TestBugCR14DeletedSnapshotSkipped.java

h3. Mechanism
{{QuotaRepairTask.recalculateSnapshotDbPendingDeleteQuota}} recomputes {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} from the active DB checkpoint and from the snapshot DBs on the repaired buckets' snapshot chains, but it skips every snapshot whose status is not {{SNAPSHOT_ACTIVE}} ({{loadActiveSnapshot}} on this commit; the patch of HDDS-17241, which lists the snapshots from the checkpoint's snapshot table instead of the live chain, keeps the same filter), so deleted snapshots are not scanned.

Deleting a snapshot ({{OMSnapshotDeleteRequest}}) only sets its status to {{SNAPSHOT_DELETED}}. Its deleted keys, deleted directories and renamed entries stay in its DB until {{SnapshotDeletingService}} moves them to the next active snapshot or to the active DB ({{OMSnapshotMoveTableKeysRequest}}, which also removes them from the deleted snapshot's DB) and then purges the snapshot. None of these requests changes the bucket's counters: an entry is charged to {{snapshotUsed*}} from the key delete until {{OMKeyPurgeRequest}} or {{OMDirectoriesPurgeRequestWithFSO}} purges it. So while a deleted snapshot still holds entries, they are pending and charged, and nothing else in the repair counts them. The deleted snapshot's DB is still available ({{OmSnapshotManager.getSnapshot(UUID)}} does not require an active snapshot, and {{SnapshotDeletingService}} also opens deleted snapshots).

h3. Trigger
* A snapshot of the bucket that was deleted and still holds deleted keys or directories of the bucket. That lasts until {{SnapshotDeletingService}} has moved them out, at least one run of the service, longer for large snapshots (the moves are batched) and while the next snapshot in the chain is also deleted (the service then skips the snapshot).
* A quota repair of that bucket ({{ozone repair om quota start}}, admin only) in that window.

h3. Impact
* {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} of the bucket end too low by what the deleted snapshot still holds (reproduced: 0 and 0 where 3000 and 1 are correct). The value is written to RocksDB and stays until a repair runs outside the window.
* By the code, not run: when those entries are purged later, the purge subtracts them again, so the counters stay too low (negative if nothing else is pending), and the quota checks, which use used plus snapshotUsed, accept writes beyond the bucket's quota by that amount.

h3. Reproduction
[^TestBugCR14DeletedSnapshotSkipped.java] goes into {{hadoop-ozone/ozone-manager}} (target package and Maven command in the class comment) and passes on unmodified source while the bug is present. It deletes a key, takes a snapshot with the real {{OMSnapshotCreateRequest}} (which moves the deleted key's row from the active DB into the snapshot DB), deletes the snapshot with the real {{OMSnapshotDeleteRequest}}, then runs {{QuotaRepairTask}} and applies the request it submits: the bucket ends at 0 and 0 although the row is still in the deleted snapshot's DB. A control with the snapshot left active ends at 3000 and 1.

h3. Patch
This patch should land together with or after the one of HDDS-17243. Without it, the newly scanned deleted snapshots are read in full, every bucket's copies included: that adds the overcount of HDDS-17243 for the deleted snapshots too, and a long scan of a snapshot that {{SnapshotDeletingService}} is about to purge can hold up that purge, which closes the snapshot DB and takes its lock while the repair still has it open, and with it the OM's double buffer flush until the scan ends (argued from the code, not run). With the prefix scan of HDDS-17243 the scan of an emptied deleted snapshot is near instant.

[^CR-14-quota-repair-scans-deleted-snapshots-not-yet-purged.patch] applies on top of the patch of HDDS-17241 (against 20d95ed37fe35013b5320713665db03c06ee36fa plus that patch); the two also apply to master at 64d4a791502. It does not conflict with the patch for pending deletes of other buckets counted from snapshot DBs (filed separately): either applies on top of the patch of HDDS-17241 alone or together with the other, in either order.

The status check is removed, so the repair scans every snapshot of the bucket that is in the checkpoint's snapshot info table, active or deleted. A purged snapshot is gone from the table and is not scanned, as before; one purged after the checkpoint fails the run through the check HDDS-17241 adds, like any other snapshot change during the scan. This does not count an entry twice. A move takes the entries out of the deleted snapshot's DB when it writes them to the destination, so a move completed before the repair is counted once, at the destination. The two writes are separate RocksDB commits, but the move updates {{lastTransactionInfo}} of both snapshots in the snapshot info cache when it is applied, so a repair that overlaps a move fails through the check HDDS-17241 adds.

Covered by {{testQuotaRepairCountsDeletedSnapshotsNotYetPurged}} in the existing {{TestQuotaRepairTask}}: three snapshots, the first two deleted. The first still holds its key (its next snapshot is deleted too), and the second has already moved its key to the third, active one with the real {{OMSnapshotMoveTableKeysRequest}}, so the test checks both the missing count and that a moved entry is counted once. On top of the HDDS-17241 patch it fails without the change ("expected: <3000> but was: <2000>"); with it, {{TestQuotaRepairTask}} and {{TestOMQuotaRepairRequest}} (22 tests) pass, {{TestOzoneRepairShell}} on a mini cluster passes with both snapshot scan patches applied, and checkstyle is clean. With the patch the reproduction's deleted snapshot test fails because the bucket is repaired to 3000 and 1, and its control passes.

Found during review of a fix for the OM quota repair, as part of the TLA+ verification effort under HDDS-15926, on commit 20d95ed37fe35013b5320713665db03c06ee36fa. Related to HDDS-17241, whose fix changes the same loop. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org