[jira] [Updated] (HDDS-17004) Streaming block read does not try another replica after a client side checksum mismatch

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17004:
------------------------------
    Attachment: TestBugCR1StreamReadChecksumNoFailover.java

> Streaming block read does not try another replica after a client side checksum mismatch
> ---------------------------------------------------------------------------------------
>
>                 Key: HDDS-17004
>                 URL: https://issues.apache.org/jira/browse/HDDS-17004
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Major
>         Attachments: CR-1-fail-over-streaming-read-on-checksum-mismatch.patch, TestBugCR1StreamReadChecksumNoFailover.java
>
>
> h3. Mechanism
> With {{ozone.client.verify.checksum=true}} (default) {{StreamBlockInputStream.StreamingReader.onNext}} verifies each ReadBlock response and fails the stream with {{OzoneChecksumException}} on a mismatch. The reader surfaces it as {{IOException("Streaming read failed")}} wrapping an {{ExecutionException}}. {{ConnectionFailureUtils.unwrapCause}} only stops at {{StorageContainerException}} and {{TimeoutIOException}}, and {{StreamBlockInputStream.handleExceptions}} (sequential reads) and {{handlePreadException}} (positioned reads) retry only those and connectivity errors. So the checksum error is rethrown and no other replica is tried. On the sequential path the failed reader is also kept, so later sequential reads of other ranges on the same stream fail with the same error until the stream is unbuffered or reopened.
> The non streaming path validates checksums inside the per datanode loop of {{XceiverClientGrpc.sendCommandWithRetry}} and moves on to the next replica for that request. Datanodes do not verify checksums for client reads by default, so silent corruption on one replica reaches the client as successful responses with bad bytes.
> h3. Trigger
> # {{ozone.client.stream.readblock.enable=true}}, a RATIS/THREE key.
> # Silent corruption of the block data on one replica (one byte flipped in place).
> # Read the key with a client whose stream starts on that replica.
> Which replica a stream starts on follows the order OM returns ({{ozone.network.topology.aware.read=true}} by default). Replicas at the same distance are shuffled per lookup, so a client hits the corrupt replica first on every read only when it is strictly the closest one (for example on the same host), and otherwise on a share of the reads.
> h3. Impact
> * A streaming read that starts on the corrupt replica fails, although two healthy replicas exist. The non streaming read of the same key succeeds.
> * No wrong data is returned.
> * Only with streaming block reads enabled (off by default).
> * On the current code the failure appears as a false EOF ({{Inconsistent read ... numBytesRead=-1}}) instead of the checksum error, because of HDDS-16680: the error arrives while {{StreamingReader.poll}} waits and is turned into end of stream before the retry decision. The fix for this issue depends on HDDS-16680 (apache/ozone PR #11388): only with both does the read succeed.
> h3. Reproduction
> PASS, deterministic in a mini cluster, unmodified source. [^TestBugCR1StreamReadChecksumNoFailover.java] ({{hadoop-ozone/integration-test}}) writes a 32 KiB RATIS/THREE key on three datanodes, flips one byte of the block file on the first pipeline node, then reads the key 5 times without and 5 times with streaming block reads. All datanodes have the client's address there, so topology aware read puts the corrupted node first for every read. A passing test means the defect is present. Results:
> ||Code||Non streaming||Streaming||Streaming failure||
> |unmodified|5/5|0/5|false EOF|
> |with the attached patch|5/5|0/5|false EOF (HDDS-16680)|
> |with PR #11388 (HDDS-16680)|5/5|0/5|{{Streaming read failed}} caused by {{OzoneChecksumException}}|
> |with both|5/5|5/5|none, each read skips the corrupt replica once|
> The short circuit read path appears to have the same gap ({{BlockInputStream}} rereads the same local file after a checksum error until its retries are used up); this is from code reading only, was not run and is not covered by the attached patch.
> h3. Patch
> [^CR-1-fail-over-streaming-read-on-checksum-mismatch.patch], against 1fb64c8e499161d1452ae6948ffd461fc9d2c302. It also applies to master at fd2f6092c6d, and together with PR #11388.
> {{ConnectionFailureUtils.unwrapCause}} (used only by {{StreamBlockInputStream}}) also returns an {{OzoneChecksumException}} found in the cause chain. On such an error the sequential path ({{handleExceptions}}) and the positioned path ({{readPositioned}}) release the failed reader and skip that replica, then open a stream on the next replica, as the non streaming path does per request. The skip lasts for one positioned read, or for one refill of the sequential read buffer (the set is cleared when the buffer is refilled). This cannot loop: within that scope every checksum failure either adds a replica not tried yet or rethrows, and a refill that succeeds returns data that the read then consumes. A checksum mismatch is a fault of the data in one range, not of the datanode, so the replica is not added to the stream's {{failedStreamingDatanodes}}: reads of other ranges, and later reads of the same range, start again from the first replica. There is no retry sleep, no block location refresh and no use of the read retry limit ({{ozone.client.max.read.retries}}) for this case, on either path; it is bounded by the number of replicas. When every replica fails the check, the read throws {{IOException("Streaming read failed")}} with the {{OzoneChecksumException}} of the last replica tried as its root cause (through an {{ExecutionException}}), the same exception as today, and later reads of other ranges still work.
> Covered in the existing unit {{TestStreamBlockInputStream}} with three mocked replicas, each case for sequential and positioned reads: {{testChecksumMismatchFailsOverToAnotherReplica}} (first replica corrupt, the read returns the data), {{testChecksumMismatchOnAllReplicasSurfacesChecksumError}} (all replicas corrupt, {{OzoneChecksumException}} is the root cause), {{testChecksumMismatchInOneRangeDoesNotAffectOtherRanges}} (offset 0 corrupt on all replicas, a read of offset 2 on the same stream returns the data) and {{testChecksumSkipDoesNotUseReadRetry}} (first replica corrupt, second answers CONTAINER_NOT_FOUND, one read retry allowed, the read returns the data from the third). The responses arrive synchronously, so the tests do not depend on HDDS-16680. Without the change the first fails with {{IOException: Streaming read failed}} caused by {{OzoneChecksumException}}, and the third fails on its sequential half because the failed reader is kept. With it {{TestStreamBlockInputStream}} (33 tests), {{TestConnectionFailureUtils}} (21), {{TestBlockInputStream}} (17), {{TestChunkInputStream}} (10), {{TestMultipartInputStream}} (6), and in a mini cluster {{TestStreamReadDatanodeFailover}} (3), {{TestStreamRead}} (4) and the integration {{TestStreamBlockInputStream}} (3) pass, checkstyle is clean.
> Related: HDDS-16873 proposes reporting replicas that fail client side checksum verification; it assumes the client already moves to another replica.
> Found by code review of the Ozone client read retry and failover paths, as part of the TLA+ verification effort under HDDS-15926, on commit 1fb64c8e499161d1452ae6948ffd461fc9d2c302. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org