[jira] [Updated] (HDDS-16771) CompleteMultipartUpload over an hsync'ed key does not flag the hsync writer's open entry

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16771:
------------------------------
    Attachment: CR-5-mark-hsync-open-key-overwritten-by-mpu-complete.patch

> CompleteMultipartUpload over an hsync'ed key does not flag the hsync writer's open entry
> ----------------------------------------------------------------------------------------
>
>                 Key: HDDS-16771
>                 URL: https://issues.apache.org/jira/browse/HDDS-16771
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Major
>         Attachments: CR-5-mark-hsync-open-key-overwritten-by-mpu-complete.patch, TestBugCR5HsyncMpuComplete.java
>
>
> h3. Mechanism
> {{S3MultipartUploadCompleteRequest.validateAndUpdateCache}} replaces an existing file table entry without checking {{HSYNC_CLIENT_ID}} on it and without adding {{OVERWRITTEN_HSYNC_KEY}} to the hsync writer's open entry. {{OMKeyCommitRequestWithFSO}} ({{:198-213}}) and {{OMKeyCommitRequest}} do both for a normal overwrite. No code under {{request/s3/multipart/}} reads or writes hsync metadata, so every bucket layout that allows hsync is affected; it was shown on FSO.
> h3. Trigger
> {{ozone.fs.hsync.enabled}} and {{ozone.hbase.enhancements.allowed}} are true (both default false), FSO bucket.
> # Writer A creates a file through {{ofs}} / {{o3fs}} and hsyncs it.
> # Another client completes a multipart upload on the same key.
> # Writer A hsyncs after crossing a block boundary, or closes.
> h3. Impact
> Writer A's stale hsync or close is accepted and silently replaces the completed multipart object, which is moved to the deleted table. A normal overwrite in the same position rejects writer A with {{KEY_NOT_FOUND}}. At complete time writer A's hsync'ed version goes to the deleted table with its blocks, and A's accepted commit then writes a key table entry that references those same blocks, so the file that comes back points at blocks already queued for deletion (from code; block deletion was not run).
> h3. Reproduction
> PASS: [^TestBugCR5HsyncMpuComplete.java] ({{hadoop-ozone/integration-test}}), public client APIs on a mini cluster.
> h3. Suggested fix
> In {{S3MultipartUploadCompleteRequest}}, for both layouts, when {{keyToDelete}} carries {{HSYNC_CLIENT_ID}}, look up that writer's open entry and add {{OVERWRITTEN_HSYNC_KEY}} in the same transaction, and have the response persist it. Whether such a complete should be rejected instead of fencing the writer is a product question.
> h3. Patch
> [^CR-5-mark-hsync-open-key-overwritten-by-mpu-complete.patch], against 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at 9c6bee807c8.
> When the entry being replaced carries {{HSYNC_CLIENT_ID}}, the request looks up that writer's open key ({{getOpenFileName}} for FSO, {{getOpenKey}} otherwise), adds {{OVERWRITTEN_HSYNC_KEY}} to the open table cache and hands the entry to the response, which writes it in the same batch. This is the handling the commit requests apply to a normal overwrite, so the writer's later hsync or close is rejected with {{KEY_NOT_FOUND}} and {{OpenKeyCleanupService}} removes the flagged open key. The marking sits after the last check that can reject the complete. The response constructors gain two parameters.
> Covered by {{testOverwrittenHsyncKeyIsMarked}} in the existing {{TestS3MultipartUploadCompleteRequest}} (inherited by the FSO variant), which fails without the change because the open key lacks {{overwrittenHsyncKey}}. The multipart request and response unit suites (122 tests) and {{TestHSync}} (42 tests) pass, checkstyle is clean. Run against the fix, the attached reproduction's stale hsync and stale close are rejected with "is already deleted/overwritten". The FSO multipart mini cluster suite was not run.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org