[jira] [Updated] (HDDS-16805) Container reconciliation does not verify chunks pulled from a peer when ozone.client.verify.checksum is false on the datanode

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-16805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-16805:
------------------------------
    Attachment: MC-7-verify-peer-chunk-checksums-in-reconciliation.patch

> Container reconciliation does not verify chunks pulled from a peer when ozone.client.verify.checksum is false on the datanode
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-16805
>                 URL: https://issues.apache.org/jira/browse/HDDS-16805
>             Project: Apache Ozone
>          Issue Type: Bug
>            Reporter: Siyao Meng
>            Priority: Minor
>         Attachments: MC-7-verify-peer-chunk-checksums-in-reconciliation.patch, TestBugMC7ReconcileUnverifiedPeerChunk.java
>
>
> h3. Mechanism
> {{KeyValueHandler.reconcileChunksPerBlock}} reads chunks from a peer through a {{BlockInputStream}} that it creates with {{conf.getObject(OzoneClientConfig.class)}}, the datanode's own view of the client configuration. The client streams verify what they read only when {{ozone.client.verify.checksum}} is true, and the peer does not verify on {{ReadChunk}}. With the setting false in the datanode's configuration, a chunk that is corrupt on the peer is written into the local replica, the local merkle tree records it as healthy with the checksum from the peer's block metadata, and the block is committed with the peer's BCSID. A chunk that the local tree lists as healthy is never pulled, so this affects blocks and chunks that the replica lacks or that its own scanner marked corrupt.
> The peer's tree is only as fresh as the peer's last scan, so a chunk that went bad since then is still listed as healthy and offered for repair.
> The reconciliation design document says that the datanode validates the checksum of a chunk read from a peer before using it, and that the existing client code already does this. That holds only while the client setting is on. {{ECReconstructionCoordinator}}, the other datanode path that reads peer data through the client streams, calls {{setChecksumVerify(true)}} on its config before it opens the stream. The line in {{reconcileChunksPerBlock}} dates from HDDS-11763 (read from history).
> h3. Trigger
> # {{ozone.client.verify.checksum=false}} is in the configuration that the datanode loads, for example a site file shared with clients.
> # A replica lacks a block or chunk that a peer has, or holds a chunk that its own scanner marked corrupt.
> # On the peer that chunk is corrupt on disk and the peer has not been scanned since.
> # An administrator runs {{ozone admin container reconcile}} for the container.
> h3. Impact
> * The corrupt chunk is copied into the reconciling replica and committed. A later round against a healthy peer in the same reconcile task does not repair it, because the local tree already lists the chunk as healthy: after reconciling with the corrupt peer and then a healthy one, the block still holds the corrupt bytes (run).
> * The on demand scan at the end of the reconcile task finds the mismatch and marks the replica UNHEALTHY. In the reproduction a replica that was CLOSED and only lacked one block before the reconcile is UNHEALTHY after it (run).
> * With the default setting the corrupt chunk is rejected with a checksum mismatch and the block is repaired from the healthy peer (run, as the control).
> h3. Reproduction
> PASS, deterministic, unmodified source. [^TestBugMC7ReconcileUnverifiedPeerChunk.java] ({{hadoop-hdds/container-service}}) uses the setup of {{TestContainerReconciliationWithMockDatanodes}}: three datanodes, each a real {{KeyValueHandler}} with a real container on disk and a real on demand scanner, the real {{BlockInputStream}}, and only the static {{ContainerProtocolCalls}} transport routed to the peer's handler. There is no SCM. One replica loses a block, one peer has a byte flipped in that block and is not scanned, the other peer is healthy. The replica reconciles with the corrupt peer first and the healthy peer second. With the setting false on the reconciling datanode the block is committed with the peer's BCSID, the block file equals the corrupt peer's bytes, and the replica is UNHEALTHY after the scan. With the setting true the block file equals the healthy bytes and the replica stays CLOSED. A passing test means the defect is present.
> h3. Patch
> [^MC-7-verify-peer-chunk-checksums-in-reconciliation.patch], against 1e528b97aed3eef1be6846f1f0767c1ef98a883a. It also applies to master at 8ca44914771.
> {{reconcileChunksPerBlock}} calls {{setChecksumVerify(true)}} on the {{OzoneClientConfig}} that it passes to the stream, as {{ECReconstructionCoordinator}} does. {{getObject}} returns a new object on every call, so no other code sees the flag. Nothing changes with the default setting. With the setting false, reconciliation now computes checksums over what it pulls, and a chunk that fails ends the repair of that block from that peer, which is the existing handling of a chunk that cannot be pulled. The next peer is then tried.
> Covered by {{testContainerReconciliationVerifiesPeerDataWhenClientChecksumVerifyDisabled}} in the existing {{TestContainerReconciliationWithMockDatanodes}}. Without the change it fails because the data checksum of the replica after reconcile and scan is not the healthy one. With it the unit suites of the {{keyvalue}}, {{ozoneimpl}} and {{checksum}} packages in {{hadoop-hdds/container-service}} pass (1124 tests, 18 skipped by the suites), checkstyle is clean, and the case of the reproduction with the setting false fails because the block now holds the healthy bytes. Not run on a mini cluster.
> Found by TLA+ model checking and code review of the container reconciliation paths in the datanode and SCM under HDDS-15926, on commit 1e528b97aed3eef1be6846f1f0767c1ef98a883a. Checked against HDDS issues and apache/ozone pull requests for duplicates before filing. The attached patch is a proposal for review. Generated with Specula (Claude Opus 5.5).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org