[jira] [Updated] (HDDS-16769) OMKeyRenameResponseWithFSO reads the bucket ID at flush time and terminates the OM if the bucket was deleted

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16769:
------------------------------
    Attachment: TestBugMC5RenameFlushBucketId.java

> OMKeyRenameResponseWithFSO reads the bucket ID at flush time and terminates the OM if the bucket was deleted
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16769
>                 URL: https://issues.apache.org/jira/browse/HDDS-16769
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Major
>         Attachments: MC-5-carry-bucket-id-in-fso-rename-response.patch, TestBugMC5RenameFlushBucketId.java
>
>
> h3. Mechanism
> {{OMKeyRenameResponseWithFSO.addToDBBatch}} calls {{getVolumeId}} and {{getBucketId}} on the double buffer flush thread instead of carrying the IDs resolved at apply time. Both are cache aware, so they see requests applied after the rename. The lookup was added so that rename updates the parent directory's modification time (HDDS-7382).
> The client reply for a write does not wait for the flush, and {{OMBucketDeleteRequest}} does not wait for unflushed responses of the bucket.
> h3. Trigger
> An FSO rename is applied and acknowledged. Before the flush thread reaches its response, everything in the bucket is deleted and the bucket is deleted, and optionally recreated under the same name.
> h3. Impact
> * Bucket deleted: {{getBucketId}} throws {{BUCKET_NOT_FOUND}} and {{OzoneManagerDoubleBuffer.addToBatch}} terminates the OM with exit status 1. A restart replays the Ratis log and recovers.
> * Bucket recreated: the parent directory rows are written under the new bucket ID while still carrying parent IDs of the old bucket's tree, and stay in {{directoryTable}}. No client visible effect of these stray rows was observed.
> h3. Reproduction
> PASS, 2 of 2 runs, with client requests against one real OM while the flush is held by {{OzoneManagerDoubleBuffer#pause}}: [^TestBugMC5RenameFlushBucketId.java] ({{hadoop-ozone/ozone-manager}}). Two concurrent designs without the hook did not hit the window.
> h3. Suggested fix
> Resolve {{volumeId}} and {{bucketId}} in {{OMKeyRenameRequestWithFSO.validateAndUpdateCache}}, which already has both under the bucket lock, and pass them to the response, as HDDS-6934 did for {{OMKeysDeleteResponseWithFSO}} after the same race crashed an OM. {{OMKeyRenameResponseWithFSO}} is the only response that still looks up these IDs at flush time.
> h3. Patch
> [^MC-5-carry-bucket-id-in-fso-rename-response.patch], against 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at c99d291e863.
> {{OMKeyRenameResponseWithFSO}} takes {{volumeId}} and {{bucketId}} as constructor arguments and {{addToDBBatch}} uses them; the flush time lookups are removed. {{OMKeyRenameRequestWithFSO.renameKey}} already has both under the bucket lock and passes them on. The parent rows are then always written under the bucket the rename was applied to; a later bucket delete removes them in log order as usual.
> Covered by {{testAddToDBBatchAfterBucketDeleted}} in the existing {{TestOMKeyRenameResponseWithFSO}}, which fails without the change with {{BUCKET_NOT_FOUND}} thrown from {{addToDBBatch}} (the exception {{OzoneManagerDoubleBuffer.addToBatch}} turns into a terminate). The rename, double buffer and {{TestOMSnapshotCreateRequest}} unit suites pass (62 tests), checkstyle is clean.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org