[jira] [Updated] (HDDS-17241) Quota repair can leave bucket snapshotUsedBytes negative when a snapshot purge commits after the repair's checkpoint and before the snapshot scan

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17241:
------------------------------
    Attachment: CR-11-fail-quota-repair-when-snapshot-changes-after-checkpoint.patch

> Quota repair can leave bucket snapshotUsedBytes negative when a snapshot purge commits after the repair's checkpoint and before the snapshot scan
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17241
>                 URL: https://issues.apache.org/jira/browse/HDDS-17241
>             Project: Apache Ozone
>          Issue Type: Bug
>    Affects Versions: 2.3.0
>            Reporter: Siyao Meng
>            Priority: Major
>         Attachments: CR-11-fail-quota-repair-when-snapshot-changes-after-checkpoint.patch, TestBugCR11QuotaRepairSnapshotPurgeSkew.java
>
>
> h3. Mechanism
> {{QuotaRepairTask.repairTask}} takes a checkpoint of the active OM DB; the bucket baselines and all active table counts come from it. {{recalculateSnapshotDbPendingDeleteQuota}} instead opens each snapshot DB live through {{OmSnapshotManager.getSnapshot}} and counts its current {{deletedTable}} and {{deletedDirTable}}. The repair sends the computed value minus the baseline for {{snapshotUsedBytes}} and {{snapshotUsedNamespace}}, and {{OMQuotaRepairRequest.updateBucketInfo}} adds it to the current bucket.
> A request that removes entries from a snapshot DB and lowers the bucket's {{snapshotUsed*}} in the same transaction ({{OMKeyPurgeRequest}} or {{OMDirectoriesPurgeRequestWithFSO}} with a snapshot table key, as sent by the deep clean of the deleting services) and commits after the checkpoint but before that snapshot is scanned is then subtracted twice: the scan no longer sees the entries, the baseline still has them, and the live bucket has already been lowered. A purge applied after both deleted tables of that snapshot have been read is harmless. The same undercount happens when a snapshot that is active in the checkpoint is deleted, has its entries moved to the active DB and is purged before the scan reaches it: the scan walks the live snapshot chain, so it skips that snapshot, and the moved entries are in neither the checkpoint's active tables nor any scanned snapshot. Nothing serializes the repair with the deleting services, and the counters are not clamped, so {{OMKeyRequest.checkBucketQuotaInBytes}} and {{checkBucketQuotaInNamespace}} (which use used plus snapshotUsed) see the bucket that much emptier.
> h3. Trigger
> * A bucket with a snapshot that still holds deleted keys or directories that are not yet deep cleaned.
> * A quota repair ({{ozone repair om quota start}}, admin only).
> * A deep clean purge of that snapshot, or its move and purge after a snapshot delete, committing between the checkpoint and the scan of that snapshot. The window covers opening the checkpoint DB, reading the bucket table and scanning the snapshots that come earlier in the scan.
> h3. Impact
> * {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} of the bucket end below their true value by what the purge removed, negative if nothing else is pending (reproduced: -3000 and -1 where 0 and 0 are correct). The value is written to RocksDB and stays until a repair runs without the race.
> * By the code, not run: quota checks undercount the bucket by the same amount, so writes beyond the quota are accepted.
> h3. Reproduction
> [^TestBugCR11QuotaRepairSnapshotPurgeSkew.java] goes into {{hadoop-ozone/ozone-manager}} (target package and Maven command in the class comment) and passes on unmodified source while the bug is present. It deletes a key into a snapshot and runs {{QuotaRepairTask}} with the request it submits captured and applied. The purge is a real {{OMKeyPurgeRequest}} with the snapshot table key; the only control is that it commits on the repair's first call to {{getOmSnapshotManager()}}, which comes from the snapshot scan after the checkpoint. A control with the purge committed before the repair ends at 0 and 0.
> h3. Patch
> [^CR-11-fail-quota-repair-when-snapshot-changes-after-checkpoint.patch], against 20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at 41d95dd3286.
> The repair now lists the snapshots of each repaired bucket from the checkpoint's {{snapshotInfoTable}} instead of the live snapshot chain, so a snapshot removed from the chain after the checkpoint is still visited. Every request that changes a snapshot DB or removes a snapshot (key and directory purge, snapshot moves, snapshot delete and purge) already updates the snapshot's {{lastTransactionInfo}} in the snapshot info cache, or removes the entry, when it is applied. After scanning a snapshot DB the repair compares the live value with the one from the checkpoint and fails the run ("Snapshot ... changed after the quota repair checkpoint was taken, run the quota repair again") instead of submitting deltas that mix two points in time; a snapshot that can no longer be opened fails the run as well. A change applied after that comparison cannot have been seen by the scan, and a change flushed before the checkpoint is in both, so neither fails the run. The run fails whenever a snapshot of a repaired bucket changed between the checkpoint and the end of its scan, which includes some changes that would not have skewed the result; rerunning is safe. Deep cleaning is on by default, so on a busy cluster a long repair can fail several times in a row; {{--buckets}} narrows the window to the snapshots of those buckets. Clamping at zero would hide the symptom but still be wrong when other deletes are pending. A possible follow up is to pause the deleting services for the scan instead, by holding {{BOOTSTRAP_LOCK}} from before the checkpoint until the snapshot scan ends, together with a double buffer flush, as {{OMDBCheckpointServlet}} does; that always converges but stalls deletion and follower bootstrap for the length of the scan.
> Covered in the existing {{TestQuotaRepairTask}}: {{testQuotaRepairFailsWhenSnapshotPurgedAfterCheckpoint}} fails without the change ("expected: 0L but was: -3000L"), {{testQuotaRepairFailsWhenSnapshotLeavesChainAfterCheckpoint}} fails without it ("expected: 3000L but was: 0L"), and {{testQuotaRepairSucceedsWhenSnapshotPurgedBeforeCheckpoint}} checks that a purge flushed before the repair does not fail the run. With the change, the quota repair, key and directory purge and snapshot move suites (46 tests) pass, including the existing {{testQuotaRepairSnapshotDbDeletedTableQuota}} for the case without a race, as does {{TestOzoneRepairShell}} on a mini cluster, and checkstyle is clean. With the patch the race test of the reproduction fails because the repair run fails, and its control passes.
> Found by code review of the OM volume, bucket and snapshot lifecycle paths, as part of the TLA+ verification effort under HDDS-15926, on commit 20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org