[jira] [Updated] (HDDS-17290) Close on an interrupted thread fails but OM still commits the key, so a small encrypted file or an interrupted write becomes empty or truncated

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17290:
------------------------------
    Affects Version/s: 2.2.1
                       2.0.0
                       2.1.2
                       2.1.0
                       2.1.1
                       2.2.0

> Close on an interrupted thread fails but OM still commits the key, so a small encrypted file or an interrupted write becomes empty or truncated
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17290
>                 URL: https://issues.apache.org/jira/browse/HDDS-17290
>             Project: Apache Ozone
>          Issue Type: Bug
>    Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
>            Reporter: Siyao Meng
>            Priority: Major
>         Attachments: CR-10-do-not-commit-key-on-interrupted-close.patch, TestBugCR10InterruptedCloseStillCommits.java
>
>
> h3. Mechanism
> {{KeyOutputStream.write}} rejects the call with {{InterruptedIOException}} ("Write aborted. Interrupted waiting for KeyOutputStream semaphore") when the thread is interrupted, because {{KeyOutputStreamSemaphore.acquire}} fails. Unlike the other write failures of {{KeyOutputStream}}, this one does not mark the stream closed, and {{acquire}} sets the interrupt flag again. A later {{close()}} therefore runs {{KeyOutputStream.closeInternal}} on the interrupted thread. If the stream holds no data in an open block, nothing in the close has to go to the datanodes, so nothing fails on the interrupt, and {{commitKey}} sends the commit with the bytes in the blocks already filled and closed (none, or only the full blocks). Hadoop IPC usually sends the request, then fails the wait for the response with "Call interrupted". The caller gets that exception, and OM applies the commit.
> For a key in an encrypted bucket the client writes through {{CryptoOutputStream}}, which keeps up to its buffer size (8 KiB by default) and hands it to the {{KeyOutputStream}} only on flush or close. Hadoop's {{CryptoOutputStream.close}} closes the inner stream even when its final flush failed (since HADOOP-18621). So a writer whose {{write}} calls all returned normally hits the same path as soon as its thread is interrupted before {{close()}}.
> h3. Trigger
> Encrypted bucket (Java client, ofs/o3fs, or any other {{OzoneOutputStream}} user):
> # Write a key or file smaller than the crypto buffer (8 KiB), or one whose data past its last full block is smaller than that, with no flush since the last full block. All {{write}} calls return normally.
> # The writer thread is interrupted before or during {{close()}}.
> Any bucket with Ratis replication:
> # Create a key, and the thread is interrupted before the first {{write}} has passed any byte to the stream, or when the data written so far ends exactly at a block boundary.
> # {{write}} fails with "Write aborted", and the stream is then closed, for example by try with resources.
> Thread interrupts of writer threads are routine in frameworks (task cancellation, {{Future.cancel(true)}}, {{ExecutorService.shutdownNow}}); the reproduction sets the interrupt flag directly. If the stream holds data in an open block, including data pushed there by a flush, the close fails writing it and nothing is committed (run). Erasure coded keys are not affected: {{ECKeyOutputStream.close}} fails with "Interrupted while adding stripe to queue" before the commit (run, empty and 16 byte keys). Datastream keys are not affected: a failed write there fails the stream (from the code). The S3 gateway form of this is HDDS-17272.
> h3. Impact
> * {{close()}} throws {{InterruptedIOException}}, yet the key is committed with size 0, or with only its full blocks, and replaces the previous content. A new key or file appears empty or truncated. Run on a mini cluster: empty commits for the Java client and ofs on an encrypted bucket and for the rejected write case on a plain bucket; truncated commits for both buckets as below.
> * The commit can keep only part of the data. Data in blocks the stream has already filled and closed does not have to be written again at close, so a key of one block plus 100 bytes on an encrypted bucket was committed with the block only (run with a 1 MiB block size). With the default 256 MiB block size this applies to any encrypted file whose data past the last full block is less than 8 KiB: a 256 MiB plus 100 byte key was committed with 268435456 bytes (run with the default block, chunk and stream buffer sizes). The same happened on a plain bucket when the write after a full block was rejected.
> * The commit becomes visible shortly after {{close()}} has thrown, so a check right after the failure can still see the old content.
> * Affects 2.0.0 and later: before HDDS-11184 added the semaphore, an interrupted write was accepted into the buffer and the close then failed on datanode I/O (from the code, not run).
> h3. Reproduction
> PASS on unmodified source. [^TestBugCR10InterruptedCloseStillCommits.java] ({{hadoop-ozone/integration-test}}) starts one MiniOzoneCluster with three datanodes and a {{JavaKeyStoreProvider}} as KMS, with a 1 MiB block size. Each test writes V1, then writes new content to the same key with the writer thread interrupted.
> * {{encryptedKeyCommittedEmptyAfterFailedClose}} (Java client) and {{encryptedOfsFileCommittedEmptyAfterFailedClose}} (ofs), 16 bytes: {{close()}} fails with "Call interrupted", then the key has size 0.
> * {{encryptedKeyTruncatedToFullBlockAfterFailedClose}}, one write of 1 MiB plus 100 bytes: {{close()}} fails with "Call interrupted", then the key holds the first 1 MiB only.
> * {{plainKeyCommittedEmptyAfterInterruptedWrite}} and {{plainKeyTruncatedAfterInterruptedWrite}} (the rejected write follows a full 1 MiB block): the write fails with "Write aborted", the close with "Call interrupted", then the key has size 0 or 1 MiB.
> * {{plainKeyKeepsPreviousContentAfterFailedClose}} (control): the accepted write leaves data in an open block, the close fails with "Failed to write chunk ... into block", and V1 stays.
> A passing test means the defect is present. With the patch all tests except the control fail because V1 stays, and the control passes.
> h3. Patch
> [^CR-10-do-not-commit-key-on-interrupted-close.patch], against b10a7d427bdf83807a7e48a1be507d3b02d5a503. It also applies to master at 1ff29ee6b86, alone and together with the patches of HDDS-17271 and HDDS-17272.
> {{KeyOutputStream.closeInternal}} throws {{InterruptedIOException}} instead of committing when the closing thread is interrupted. The check runs after the pre-commits and before {{commitKey}}. A close on a thread that is already interrupted no longer sends the commit, and the open key is left to the open key cleanup. The caller already got an exception in this case. Uninterrupted closes are unchanged. Failing the stream on a rejected write would also cover a caller that clears the interrupt flag or closes the stream from another thread, which still commits the bytes written before the rejected write, but it changes {{write}} for callers that keep using the stream and touches the lines changed by HDDS-17271.
> Covered by {{testInterruptedCloseOfEncryptedKeyDoesNotCommit}} (key stream wrapped in a real {{CryptoOutputStream}}) and {{testCloseAfterInterruptedWriteDoesNotCommit}} in the existing {{TestOzoneClient}}, which check that the key keeps its previous content. Without the change they fail with {{expected: 14L but was: 0L}} and "Expecting code to raise a throwable". With it the {{hadoop-ozone/client}} unit tests (340), {{TestOzoneAtRestEncryption}} (36) and {{TestHSync}} (43) on a mini cluster pass, and checkstyle is clean. With this patch alone, the reproduction of HDDS-17272 also no longer commits.
> Found by code review of the S3 gateway object put and key commit paths, as part of the TLA+ verification effort under HDDS-15926, on commit b10a7d427bdf83807a7e48a1be507d3b02d5a503. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org