[jira] [Updated] (HDDS-16770) FSO rename response serializes the live cached OmBucketInfo, corrupting bucket usage after crash and replay

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16770:
------------------------------
    Attachment: CR-2-copy-bucket-info-into-fso-rename-response.patch

> FSO rename response serializes the live cached OmBucketInfo, corrupting bucket usage after crash and replay
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16770
>                 URL: https://issues.apache.org/jira/browse/HDDS-16770
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Major
>         Attachments: CR-2-copy-bucket-info-into-fso-rename-response.patch, TestCR2RenameBucketAlias.java
>
>
> h3. Mechanism
> {{OMKeyRenameRequestWithFSO.setModificationTime}} puts a new {{OmBucketInfo}} into the bucket table cache, and {{renameKey}} hands the same instance to {{OMKeyRenameResponseWithFSO}} ({{:353}}), which serializes it on the double buffer flush thread. Later requests mutate that cached instance in place ({{OMKeyRequest.getBucketInfo}} returns the cached reference) before the rename's batch is written.
> The bucket row is therefore committed in a batch whose {{TransactionInfo}} is still below the later request's index, but already contains that request's {{usedNamespace}} / {{usedBytes}} delta.
> h3. Trigger
> A rename whose source or destination parent is the bucket root (a rename between directories carries no bucket in the response), followed by a request that changes bucket usage, such as a key commit, applied while the flush thread is still working through the batch that holds the rename, then an OM crash before the later request's batch is committed.
> h3. Impact
> On restart Ratis replays the later request on top of a bucket row that already includes it. With a tight namespace quota an acknowledged commit is rejected with {{QUOTA_EXCEEDED}} on replay and the file is missing on that OM. Otherwise bucket usage is double counted until a quota repair is run.
> h3. Reproduction
> PASS at component level: [^TestCR2RenameBucketAlias.java] ({{hadoop-ozone/ozone-manager}}) uses real RocksDB, table caches, the double buffer flush thread and the request classes, with the flush timing controlled. The crash and the replay are driven by the test, not by a real Ratis restart.
> h3. Suggested fix
> Give the response its own snapshot: pass {{omBucketInfo.copyObject()}} (when non null) to the response, matching the other FSO requests. HDDS-16125 fixed the same aliasing in {{OMQuotaRepairRequest}}.
> h3. Patch
> [^CR-2-copy-bucket-info-into-fso-rename-response.patch], against 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at c99d291e863.
> One line: {{renameKey}} passes {{omBucketInfo.copyObject()}} (or null when the rename does not touch the bucket) to the response. The copy is taken under the bucket lock, so no later request can have touched the object yet. This is what the other FSO key requests already do.
> Covered by {{testFlushedBucketExcludesLaterUsageChange}} in the existing {{TestOMKeyRenameRequestWithFSO}}: it applies a root level rename, changes {{usedNamespace}} on the cached bucket as a later request would, flushes the rename response and reads the bucket row from RocksDB. Without the change it fails with "expected: 0L but was: 1L". The rename and double buffer unit suites pass (44 tests), checkstyle is clean. A bucket that already double counted needs the existing quota repair.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org