[jira] [Created] (HDDS-16793) Quota repair adds deletedTable entries without the committed flag to the pending deletion usage of the bucket, and the purge never releases them

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16793:
---------------------------------

             Summary: Quota repair adds deletedTable entries without the committed flag to the pending deletion usage of the bucket, and the purge never releases them
                 Key: HDDS-16793
                 URL: https://issues.apache.org/jira/browse/HDDS-16793
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: CR-2-count-only-committed-deleted-keys-in-quota-repair.patch, TestBugCR2QuotaRepairCountsUncommittedDeletedKeys.java

h3. Mechanism
{{QuotaRepairTask.countDeletedKey}} (HDDS-15204) adds the replicated size of every key info in the {{deletedTable}} entries of a bucket to {{snapshotUsedBytes}}, and one unit per key info to {{snapshotUsedNamespace}}. The requests that maintain these counters follow a narrower rule (HDDS-13756). Only a key that leaves the key table after commit is added to them, and it is marked with {{OmKeyInfo.withCommittedKeyDeletedFlag(true)}}: by {{AbstractOMKeyDeleteResponse}} for DeleteKey and DeleteKeys, by {{OMKeyCommitRequest}} for the key it overwrites, and by {{OMDirectoriesPurgeResponseWithFSO}} for sub files. When the entry is purged, {{KeyDeletingService}} and {{OMKeyPurgeRequest}} release only key infos that have the flag.

Every other key info in {{deletedTable}} has no flag and was never added to the counters: the key that {{S3MultipartUploadCompleteRequest}} overwrites, the parts of an aborted multipart upload, an overwritten part and the unused parts of a completed upload, expired open keys ({{OMOpenKeysDeleteRequest}}), the uncommitted blocks of a key commit, and entries written before HDDS-13756. A repair that runs while such an entry is waiting adds an amount to the counters that no purge removes.

By history, not run on older code: the recount of {{deletedTable}} was added by HDDS-15204 and is in no release tag up to 2.2.1, so only unreleased master is affected.

h3. Trigger
# A bucket has a {{deletedTable}} entry without the committed flag. Run with a key that is overwritten by a completed multipart upload. The other sources listed above are from code reading, not run.
# Quota repair runs for the bucket before {{KeyDeletingService}} purges the entry. An admin starts it with {{ozone repair om quota start}} ({{OzoneManager.startQuotaRepair}}). The service interval is 60s by default. The entry of the overwritten key stays for as long as a snapshot references that key (run separately on the same harness, not part of the attached test: still present after 30 service runs with a snapshot taken before the overwrite).
# The entry is purged.

h3. Impact
* With {{deletedTable}} empty again, {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} of the bucket keep the size and one unit for each key info that had no flag when the repair ran. Both are part of the usage that the quota checks compare. Run: a bucket with a namespace quota of 2 that holds one key rejects a second key with {{QUOTA_EXCEEDED}}. The space quota check reads {{snapshotUsedBytes}} the same way (argued, not run).
* No data is affected. The counters stay wrong until quota repair runs again at a time when no such entry is waiting (argued, not run).

h3. Reproduction
PASS, deterministic, unmodified source. [^TestBugCR2QuotaRepairCountsUncommittedDeletedKeys.java] ({{hadoop-ozone/ozone-manager}}) drives one real OM with its {{KeyDeletingService}} through {{OzoneManagerProtocol}}, with the testing SCM block client of the OM tests and no datanodes. Repair is started with {{new QuotaRepairTask(om).repair(buckets)}}, the body of {{OzoneManager.startQuotaRepair}}. The service interval is 100 ms and the service is held with its own suspend and resume hook so that the repair runs while the entry is waiting. {{multipartOverwriteThenRepairLeavesPendingCountersStuck}} shows {{snapshotUsedBytes=805306368}} and {{snapshotUsedNamespace=1}} after the repair and unchanged after the purge, with an empty {{deletedTable}}, and the rejected second key. {{keyOverwriteThenRepairReturnsToZero}} is the control with a plain key overwrite, where the counters return to zero and the second key is accepted. A passing first test means the defect is present.

h3. Patch
[^CR-2-count-only-committed-deleted-keys-in-quota-repair.patch], against a6332fb59879b80615b9233e953d6813d03ae7a8. It also applies to master at 3a22acfaa21.

{{countDeletedKey}} counts a key info only if {{isDeletedKeyCommitted()}} is set, which is the rule the purge uses to release it. The size formula is unchanged (replicated size per key info, HDDS-16176 covers that formula). The function is shared by the scan of the active {{deletedTable}} and the scans of the snapshot {{deletedTable}}s, so both follow the rule. The patch does not change which entries carry the flag.

Covered by {{testQuotaRepairCountsOnlyCommittedDeletedKeys}} in the existing {{TestQuotaRepairTask}}: an entry with the flag, an entry without it and an entry that holds one key info of each kind (as a key commit writes when it overwrites a key and drops uncommitted blocks) must count as two keys. Without the change it fails with "expected: 2000L but was: 5000L". The existing fixtures ({{putDeletedKey}} in the test and {{OMRequestTestUtils.deleteKey}}) now write the flag, as a real delete of a committed key does. With the change {{TestQuotaRepairTask}}, {{TestOMQuotaRepairRequest}}, {{TestOMKeyPurgeRequestAndResponse}}, {{TestOMDirectoriesPurgeRequestAndResponse}} and {{TestKeyDeletingService}} (55 tests) and {{TestOzoneRepairShell}} on a mini cluster (3 tests) pass, checkstyle is clean. With the patch the first test of the reproduction fails at the counters after the repair (0 instead of 805306368), the control still passes.

Found by code review of the OM key deletion service and key purge paths, as part of the TLA+ verification effort under HDDS-15926, on commit a6332fb59879b80615b9233e953d6813d03ae7a8. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org