[jira] [Closed] (HDDS-17287) On a versioned bucket, a key commit writes back older versions whose blocks the key committed at that time no longer references

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng closed HDDS-17287.
-----------------------------

> On a versioned bucket, a key commit writes back older versions whose blocks the key committed at that time no longer references
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17287
>                 URL: https://issues.apache.org/jira/browse/HDDS-17287
>             Project: Apache Ozone
>          Issue Type: Bug
>    Affects Versions: 0.2.1, 0.3.0, 0.4.0, 0.4.1, 0.5.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.2.1, 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
>            Reporter: Siyao Meng
>            Priority: Critical
>         Attachments: MC-2-versioned-commit-keeps-older-blocks-still-in-use.patch
>
>
> h3. Mechanism
> With bucket versioning enabled, {{OMKeyRequest.prepareFileInfo}} builds the open key of an overwrite from the committed key with {{OmKeyInfo.addNewVersion(locations, false, true)}}, so the open key carries the version groups the key had at open time. {{OMKeyCommitRequest}} and {{OMKeyCommitRequestWithFSO}} write the open key with all these groups to the key table, and on a versioned bucket they never compare the older groups with the key committed at that time. If that key was deleted after the open, {{OMKeyDeleteRequest}} has already moved its blocks to {{deletedTable}}. If it was renamed, the blocks are the current version of the renamed key. If the key was replaced in between, for example by the commit of another overwrite after a delete or a rename, the key committed at that time does not reference them either, although it has the same object ID. In each case the commit writes references to blocks the committed key does not own.
> h3. Impact
> * After a delete, the key references blocks that {{KeyDeletingService}} sends to SCM for deletion while the key still lists them. Deleting the key later queues them a second time and subtracts their size from the bucket used bytes a second time.
> * After a rename, the key references the current blocks of the renamed key. Deleting the key puts them in {{deletedTable}}, so the renamed key loses its current data once the datanodes delete the blocks (the datanode deletion was not run).
> * The latest version of the committed key stays readable, because reads use only the latest version group.
> h3. Reproduction
> {{TestOMKeyCommitRequest#testCommitOnVersionedBucketAfterCommittedKeyChanged}} in the attached patch (inherited by {{TestOMKeyCommitRequestWithFSO}}) fails on an unmodified checkout. On a versioned bucket it commits a key, builds the open key of an overwrite as {{prepareFileInfo}} does, then leaves the committed key alone, seeds a deleted key in the key table cache, seeds a new object under the same name in the cache, seeds a deleted key and commits a second overwrite of the same object, or renames the key with {{OMKeyRenameRequest}} and commits a second overwrite, and finally commits the first overwrite. Every case but the first fails for both layouts:
> {noformat}
> Expecting actual:
>   [conID: 1000 locID: 100 bcsId: 0 replicaIndex: null,
>     conID: 1001 locID: 101 bcsId: 0 replicaIndex: null]
> to contain exactly (and in same order):
>   [conID: 1001 locID: 101 bcsId: 0 replicaIndex: null]
> but some elements were not expected:
>   [conID: 1000 locID: 100 bcsId: 0 replicaIndex: null]
> {noformat}
> Also run on one OM with its background services ({{OmTestManagers}}) and an SCM stub that records block deletions. With a delete before the commit, the block of the deleted version was sent to SCM while the key still listed it, and deleting the key again queued it a second time. With a rename before the commit, deleting the overwritten key put the current block of the renamed key in {{deletedTable}}. Not exercised: block deletion on datanodes, the S3 gateway and an OM HA ring.
> h3. Patch
> [^MC-2-versioned-commit-keeps-older-blocks-still-in-use.patch], against b10a7d427bdf83807a7e48a1be507d3b02d5a503. It also applies to master at 26388a80b37.
> A new {{OMKeyRequest.dropOlderBlocksNotInUse}} keeps a block of an older version of the open key only if the key committed at that time still references it (compared by {{ContainerBlockID}}, the inverse of {{filterOutBlocksStillInUse}}), and drops older versions that are left empty. The latest version holds the blocks of the open key itself and is kept as is. {{OMKeyCommitRequest}} and {{OMKeyCommitRequestWithFSO}} apply it to the open key before building the committed key. A normal versioned overwrite keeps all versions, because the committed key still references them, and an open key with a single version (no versioning, or a new key) is returned unchanged. Kept versions keep their version numbers. Dropped blocks are not queued for deletion: a delete has already queued them, a rename has moved them to another key, and a block that an earlier versioned commit already left unreferenced stays unreferenced and is not reclaimed (HDDS-16733).
> With the change the new test passes, and so do all key request suites under {{om.request.key}} with the commit response suites (413 tests) and {{TestOmBlockVersioning}} on a mini cluster (2 tests). Checkstyle is clean.
> An alternative is to reject enabling versioning on a bucket until the new versioning design (HDDS-15728) lands.
> Related: HDDS-16733 (concurrent open of an absent key on a versioned bucket drops a version) changes the same commit branch for a different case, and the two changes do not overlap. HDDS-16732 (open key cleanup on a versioned bucket) has the same cause, the open key carrying the committed versions, on the expiry path. HDDS-16792 covers a delete after a stale overwrite commit on a bucket without versioning.
> Found by TLA+ model checking and code review of the S3 gateway object put and key commit paths under HDDS-15926, on commit b10a7d427bdf83807a7e48a1be507d3b02d5a503. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org