You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/09/22 17:16:00 UTC
[jira] [Updated] (HDDS-16258) Support boundary-aware checksum verification for variable-sized chunks in StreamBlockInputStream
[ https://issues.apache.org/jira/browse/HDDS-16258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HDDS-16258:
----------------------------------
Labels: pull-request-available (was: )
> Support boundary-aware checksum verification for variable-sized chunks in StreamBlockInputStream
> ------------------------------------------------------------------------------------------------
>
> Key: HDDS-16258
> URL: https://issues.apache.org/jira/browse/HDDS-16258
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Chung-En Lee
> Assignee: Chung-En Lee
> Priority: Major
> Labels: pull-request-available
>
> *Description:*
> *Background* As part of HDDS-15857, we fixed the server-side checksum calculation in {{KeyValueHandler}} for variable-sized chunks and updated the protocol to optionally include {{chunkInfoList}} in the {{{}ReadBlockResponseProto{}}}. To keep the original patch focused and manageable, the client-side verification logic has been split into this ticket.
> *Problem* Currently, when a client actively triggers a flush or hsync, the resulting chunk size may not perfectly align with {{{}bytesPerChunk{}}}. The client-side {{StreamBlockInputStream}} incorrectly assumes all chunks (except the last one) have a fixed size when verifying checksums.
> Because of this fixed-size assumption, the client cannot correctly map the offsets for variable-sized chunks. This leads to misaligned checksum verification, causing false {{{}OzoneChecksumException{}}}s when processing chunks that are variable-sized or smaller than {{{}bytesPerChunk{}}}.
> *Proposed Solution* Update the client-side streaming read architecture to be aware of exact chunk boundaries:
> # *Boundary-Aware Verification:* In {{{}StreamBlockInputStream#onNext{}}}, if the server response contains {{chunkInfoList}} and checksum verification is enabled:
> ** Iterate through each chunk to identify the exact overlapping region between the read block and the chunk boundaries.
> ** Calculate the correct start and end checksum indices using {{bytesPerChecksum}} relative to that specific chunk's starting offset.
> # *Backward Compatibility:* If {{chunkInfoList}} is absent from the response (e.g., when reading from an older DataNode version), the client must gracefully fall back to the legacy verification path ({{{}Checksum.verifyChecksum(data, checksumData, 0){}}}) to avoid breaking upgrades.
> *Testing*
> * Update {{TestStreamBlockInputStream.java}} to validate boundary-aware verification using {{{}chunkInfoList{}}}.
> * Add integration tests in {{TestStreamRead.java}} (e.g., {{testSmallChunksWithLargeChecksum}} and {{{}testShiftedChecksumBoundaryVerification{}}}) to cover varying/small chunk sizes with larger {{{}bytesPerChecksum{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org