[jira] [Updated] (HDDS-16974) EC file checksum is not bounded by the committed block group length when a replica lists a stripe beyond it

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16974:
------------------------------
    Attachment: MC-1-bound-ec-block-checksum-by-committed-block-length.patch

> EC file checksum is not bounded by the committed block group length when a replica lists a stripe beyond it
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16974
>                 URL: https://issues.apache.org/jira/browse/HDDS-16974
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Major
>         Attachments: MC-1-bound-ec-block-checksum-by-committed-block-length.patch
>
>
> h3. Mechanism
> {{ECBlockChecksumComputer}} computes the checksum of an EC block group from the stripe checksums in the chunk list of one replica ({{ECFileChecksumHelper.getChunkInfos}}, one {{ChunkInfo}} per stripe). It is given the block group length committed in OM as {{blockLength}}, but neither mode stops at that length:
> * {{computeMd5Crc}} never reads {{blockLength}} and digests the stripe checksums of every listed stripe.
> * {{computeCompositeCrc}} counts {{blockSize}} down from {{blockLength}}, but its loop does not end when the length is used up.
> A replica can list a stripe beyond the committed length. When the PutBlock of a stripe fails on one replica, {{ECKeyOutputStream}} writes the stripe again into a new block group, and the replicas that stored it keep it in their chunk list. The data read path is bounded by the OM length, the javadoc of {{ECReconstructionCoordinator.calcEffectiveBlockGroupLen}} accepts the extra stripe for that reason, and HDDS-10985 bounded the PutBlock of reconstruction for the same state. The checksum path was not adapted.
> h3. Trigger
> # A replica of an EC block group lists a stripe beyond the length committed for that block group. Reached on a mini cluster by stopping one datanode between two stripes of a key write (rs-3-2-1024k, replica 2, last stripe shorter than one cell).
> # The file checksum of the key is requested ({{ozone sh key checksum}}, or {{FileSystem#getFileChecksum}} of ofs and o3fs).
> h3. Impact
> The key is committed and reads back correctly, only its file checksum is affected.
> * MD5MD5CRC: the checksum includes the extra stripe. Two keys with the same bytes and the same block layout got different checksums (run on the mini cluster).
> * COMPOSITE_CRC, the default of {{ozone.client.checksum.combine.mode}}: the computation does not stop at the block length. On the mini cluster the call for the affected key had not returned after 20 seconds, while keys with the same bytes and no extra stripe returned at once.
> * Nothing removes the extra stripe from the replicas (code reading, not run), so the key behaves this way for as long as the checksum code does not stop at the block length. Affected keys need no repair: with the patch the affected key returned the same checksum as the unaffected keys in both modes.
> h3. Reproduction
> No standalone reproduction is attached to this issue. The regression test in the patch, {{TestECBlockChecksumComputer.testStripeBeyondBlockLengthIsIgnored}}, covers both modes at unit level and fails on unmodified source: the MD5MD5CRC case with a different digest, the COMPOSITE_CRC case with its 30 second timeout.
> h3. Patch
> [^MC-1-bound-ec-block-checksum-by-committed-block-length.patch], against 031e76aa2c9a60b67185390b5b33499cbbb82e43. It also applies to master at c5d4f28266d.
> {{ECBlockChecksumComputer}} stops at the block length in both modes. {{computeMd5Crc}} counts the block length down by one stripe per {{ChunkInfo}} and skips what lies beyond it. The loop in {{computeCompositeCrc}} ends once nothing of the block length is left to consume. A block whose chunk list matches its length gets the same result as before: generated chunk lists over a range of cell sizes, bytes per checksum and block lengths gave identical results with and without the patch, and {{TestOzoneFileChecksum}} passes on a mini cluster (6 tests). The write path is left as it is, so the replicas keep the extra stripe, as they do today for reads and reconstruction.
> With the patch the checksum unit suites of {{hadoop-ozone/client}} pass (27 tests) and checkstyle is clean.
> Found by TLA+ model checking and code review of the client EC key write and file checksum paths under HDDS-15926, on commit 031e76aa2c9a60b67185390b5b33499cbbb82e43. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. Related but not duplicates: HDDS-10985 (resolved, the same state in reconstruction) and HDDS-16834 (open, a block without CRC bytes in COMPOSITE_CRC mode). The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org