[jira] [Created] (HDDS-16790) KeyDeletingService can purge a deleted key in one PurgeKeys request and carry its size in another, leaving the size in the bucket usage for good

["Siyao Meng (Jira)" <ji...@apache.org>]

Siyao Meng created HDDS-16790:
---------------------------------

             Summary: KeyDeletingService can purge a deleted key in one PurgeKeys request and carry its size in another, leaving the size in the bucket usage for good
                 Key: HDDS-16790
                 URL: https://issues.apache.org/jira/browse/HDDS-16790
             Project: Apache Ozone
          Issue Type: Bug
            Reporter: Siyao Meng
         Attachments: MC-3-keep-deleted-key-and-its-size-in-one-purge-request.patch, TestBugMC3SplitPurgeLeaksBucketUsage.java

h3. Mechanism
{{KeyDeletingService.submitPurgeKeysRequest}} builds PurgeKeys requests from block groups, one per version of a {{deletedTable}} entry. For every block group it adds the name of the entry to the request, but it adds the size to {{bucketPurgeKeysSize}} only for a committed version. After every block group it compares the request size with 90% of {{ozone.om.ratis.log.appender.queue.byte-limit}} and, when the limit is reached, submits the request and starts a new one. An entry that holds a committed version and a group of uncommitted blocks can therefore be named without its size in one request and with its size in the next.

{{OMKeyPurgeRequest}} and {{OMKeyPurgeResponse}} delete the entry by name and subtract from {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} of the bucket only what the same request carries. If the first of the two requests is applied and the second is not, the entry is gone and nothing subtracts its size any more.

The size based split (HDDS-13517) and the per bucket size in the request (HDDS-13764) are both first contained in release 2.1.0 (by history, not run on older code).

h3. Trigger
# A key is overwritten by a commit that leaves an allocated block uncommitted, so its {{deletedTable}} entry has the old version and a group of uncommitted blocks.
# One run of the service collects more than fits into one request. At default limits (32MB, {{ozone.key.deleting.limit.per.task}} 50000) that needs {{deletedTable}} keys of about 600 bytes on average, which means key names or paths of about that length (the average is over every entry of the run, whatever the bucket layout). It needs less when the per task limit is raised or the Ratis limit is lowered.
# The cut falls between the two block groups of that entry, with the uncommitted blocks first. The two groups are close to each other in iteration order, so only the entries right at a cut are exposed. Which of the two comes first is about even.
# The next request of the run is not applied.

For step 4 I ran this case: a snapshot of any bucket is created between the two submits. {{OMKeyPurgeRequest}} then rejects every later request of that run with {{INVALID_REQUEST}}, because the expected previous snapshot has changed. No fault is involved. Not run by me: an OM stop or a leader change between the two submits (argued, the state after the first request is the same). Also argued, not run: the request that carries the size fails with a {{ServiceException}}, which the service catches before it submits the next request, and the request that only names the entry is applied.

h3. Impact
* The bucket keeps the size of the purged entry in {{snapshotUsedBytes}} and one key in {{snapshotUsedNamespace}}. In the reproduction the bucket is empty, its deleted table is empty, and both OM and the client still report 805306368 bytes and 1 key in use.
* The leaked amount counts against the quota. With a namespace quota of 1 the first key written to the empty bucket is rejected with {{QUOTA_EXCEEDED}}.
* Nothing recomputes the counters on its own. A quota repair started by an admin does (by code reading, not run).
* No data is lost and the blocks are deleted as intended. One incident leaks the entries that sit at a cut, typically one.

h3. Reproduction
PASS on unmodified source. [^TestBugMC3SplitPurgeLeaksBucketUsage.java] ({{hadoop-ozone/ozone-manager}}) drives one real OM through {{OzoneManagerProtocol}}, without datanodes. It overwrites 20 keys so that each leaves one uncommitted block, runs {{KeyDeletingService}} once, and creates a snapshot of an unrelated bucket through the client API right after the first applied request that names an entry without its size. It then shows that the later request for that entry is rejected with {{INVALID_REQUEST}}, that the deleted table is empty, and the leaked usage and the {{QUOTA_EXCEEDED}} error described above. To keep the test small the Ratis limit is 64KB and the key names are 59000 characters long, so every block group fills its own request. A split at default limits (about 30MB of names in one run) was not run. A passing test means the defect is present.

h3. Patch
[^MC-3-keep-deleted-key-and-its-size-in-one-purge-request.patch], against a6332fb59879b80615b9233e953d6813d03ae7a8. It also applies to master at 3a22acfaa21.

{{KeyDeletingService.submitPurgeKeysRequest}} groups the block groups by deleted key name and adds one {{deletedTable}} entry per step, its name once and the sizes of all its committed versions together. The size check stays where it is, so a request can only be cut between entries. An entry that alone exceeds the limit is sent in a request of its own, as before. The request format and the way it is applied do not change. One visible side effect: {{NumKeysPurged}} is counted from the names in the request, so an entry with two reclaimable versions now counts once instead of twice. {{NumKeysSentForPurge}} and the deleted key count of the service still count versions, so the two no longer agree for such entries.

Covered by {{testPurgeKeysRequestKeepsKeyVersionsTogether}} in the existing {{TestKeyDeletingService}}, which also pins that an entry that is rewritten instead of purged carries its size in the request that updates it. Without the change it fails because the overwritten entry is spread over two requests (four requests instead of three). With it {{TestKeyDeletingService}}, {{TestOMKeyPurgeRequestAndResponse}} and {{TestKeyManagerImpl}} pass (59 tests) and checkstyle is clean. {{TestKeyPurging}} and {{TestSnapshotDeletingServiceIntegrationTest}} on a mini cluster (9 tests) pass in a run that had this patch and a second, unrelated change to {{OMKeyPurgeRequest}} applied together. With the patch the reproduction fails at its first assertion, because no request names an entry without its size any more. Buckets that already leaked are not repaired by the patch.

Found by TLA+ model checking and code review of the OM key deletion service and key purge paths under HDDS-15926, on commit a6332fb59879b80615b9233e953d6813d03ae7a8. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. Related but not duplicates: HDDS-13517 (introduced the size based split of PurgeKeys requests), HDDS-13764 (introduced the per bucket size in the request), HDDS-16184 (quota repair of the same counters). The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org