[jira] [Updated] (HDDS-16981) Ratis key write waits for its own retry handling when the data written on retry fills the replacement block

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16981:
------------------------------
    Attachment: CR-6-do-not-close-block-filled-on-retry-inside-retry-handling.patch

> Ratis key write waits for its own retry handling when the data written on retry fills the replacement block
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16981
>                 URL: https://issues.apache.org/jira/browse/HDDS-16981
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Major
>         Attachments: CR-6-do-not-close-block-filled-on-retry-inside-retry-handling.patch
>
>
> h3. Mechanism
> Since HDDS-11239 (2.0.0) the block that {{KeyOutputStream.handleRetry}} allocates to take over the unacknowledged data of a failed block is created in retry handling mode ({{BlockOutputStreamEntry.isHandlingRetry}}). Calls that find the flag set wait in {{waitForRetryHandling}} until {{KeyOutputStream.handleException}} has finished the handling of the failed block and calls {{finishRetryHandling}}.
> {{handleRetry}} writes the data again through {{handleWrite(retry=true)}}. When that data fills the new block, {{handleWrite}} calls {{handleFlushOrClose(StreamAction.FULL)}} to close it. {{handleFlushOrClose}} waits for the retry handling of the current block like any other call, but here the caller is the retry handling, so the thread waits for a signal that only it can send.
> The Javadoc of {{handleFlushOrClose}} assumes that a block written on retry "will be at max half full". That holds when the block is at least twice {{ozone.client.stream.buffer.max.size}}, and nothing enforces it.
> h3. Trigger
> # {{ozone.scm.block.size}} at or below {{ozone.client.stream.buffer.max.size}}. The defaults are 256MB and 32MB, so the default block size is not affected. Nothing validates the two settings against each other.
> # A block of a Ratis key fails while a full block of its data is unacknowledged, and a new block is allocated for the retry. If a block preallocated at key creation takes the retry, it is not in retry handling mode and the write returns.
> Run on unmodified source with mock datanodes and default buffer sizes (blocks of 1MB, 4MB, 16MB and 32MB wait, blocks of 64MB and 256MB return), and on a mini cluster with a block size of 1MB and no failure injected, where SCM closed the container of the first block before the block was sent.
> h3. Impact
> * {{write}} does not return. There is no exception and no timeout, and only an interrupt ends the wait. The stream is closed then and the key is not committed.
> * No committed data is affected.
> * Workaround: a block size above {{ozone.client.stream.buffer.max.size}}.
> h3. Reproduction
> No standalone reproduction is attached to this issue. The regression test in the patch, {{TestOzoneClient.testPutKeyRatisWhenRetriedDataFillsBlock}}, covers it with mock datanodes: without the change five of its six cases run into their timeout, the sixth is a control with a half full block.
> h3. Patch
> [^CR-6-do-not-close-block-filled-on-retry-inside-retry-handling.patch], against 031e76aa2c9a60b67185390b5b33499cbbb82e43. It also applies to master at d0454e0cffd.
> {{KeyOutputStream.handleWrite}} no longer issues the {{FULL}} action when the write is the rewrite of a retry. The full block stays the current block and is closed once the handling of the failed block is finished, through the normal wait, by the action that was being retried, by the next {{write}} or by {{close}}.
> The change takes effect only when the data written again fills the block. Besides the case above that is a retry that lands on a preallocated block, which does not wait today: there the block is closed one step later with the same result (run with mock datanodes and on the mini cluster, with and without the patch: same block lengths, key reads back).
> Skipping the wait for {{FULL}} would also end the wait, but it is not safe with concurrent calls ({{ozone.client.key.write.concurrency}} other than 1): the writer closes the retry block and moves to a next block while the handoff of the failed block is still pending. In a mock datanode run of that variant with a pending {{hsync}}, {{close}} failed with "Expected offset: 2560 expected len: 3072". With the attached patch the same run returns for both calls and the key reads back.
> The regression test checks the committed block lengths and reads the data back, for a key of exactly one block and a key of 2.5 blocks, with the failure noticed by the close of the full block or by the write that fills it, and with the replacement block failing as well. With the patch all tests of {{hadoop-ozone/client}} (165) pass, checkstyle is clean, and on a mini cluster {{TestHSync}} (42), {{TestBlockOutputStreamWithFailures}} (20), {{TestFailureHandlingByClient}} (6), {{TestFailureHandlingByClientFlushDelay}} (1), {{TestOzoneClientRetriesOnExceptions}} (2), {{TestOzoneClientRetriesOnExceptionFlushDelay}} (1) and {{TestMultiBlockWritesWithDnFailures}} (2) pass.
> The patch is independent of HDDS-16979 and HDDS-16980 and applies before or after their patches. It and the patch of HDDS-16975 both add imports and a test to {{TestOzoneClient}}, so the one merged second needs a rebase of that file.
> Found by code review of the Ratis key write path while validating fixes from the TLA+ verification effort under HDDS-15926, on commit 031e76aa2c9a60b67185390b5b33499cbbb82e43. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org