[jira] [Commented] (HDFS-17986) Replica with a corrupt meta header is not reported when a short-circuit read skips checksum verification

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-17986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18123011#comment-18123011 ] 

ASF GitHub Bot commented on HDFS-17986:
---------------------------------------

joseluisll opened a new pull request, #8789:
URL: https://github.com/apache/hadoop/pull/8789

   ### Description of PR
   
   [HDFS-17986](https://issues.apache.org/jira/browse/HDFS-17986): with short-circuit reads enabled and checksum verification disabled (as HBase does by default), a replica with a corrupt meta file header is never reported. The client fails to parse the header and falls back to a remote read without checksums, so the DataNode never opens the meta file.
   
   `DataNode#requestShortCircuitFdsForRead` now reads the meta header before passing the file descriptors. If the read fails on a finalized replica, the DataNode calls `handleBadBlock`. The descriptors are still returned, because failing the request would make the client disable short-circuit reads for the whole DataNode.
   
   This supersedes the approach in HDFS-17179 / #8310.
   
   ### How was this patch tested?
   
   **New unit test `TestShortCircuitCorruptMetaHeader`** (needs native domain sockets). Each case:
   1. Starts a MiniDFSCluster with short-circuit reads enabled and writes a file.
   2. Corrupts the meta header of one replica, either with an invalid checksum type or by truncating the meta file.
   3. Reads the file from that replica, with checksum verification on or off.
   4. Checks that the read behaves as expected and that the NameNode marks the replica corrupt.
   
   It runs with replication 1, 3, 5 and 7 (16 cases). Without the fix, the 8 cases with checksums off fail: the read succeeds but the replica is never reported. With the fix, all 16 pass.
   
   **Existing tests:** the short-circuit and corrupt-metadata suites pass locally (`TestShortCircuitLocalRead`, `TestBlockReaderFactory`, `TestShortCircuitCache`, `TestBlockReaderLocal`, `TestCorruptMetadataFile`, `TestScrLazyPersistFiles`).
   
   **End-to-end, before the fix:** HBase 2.6.7 on HDFS 3.5.0 with default settings. After corrupting the header of one block of a large HFile, HBase read all rows without errors on every scan, while `hdfs fsck` kept reporting the file as HEALTHY. Details are in the JIRA.
   
   ### For code changes:
   
   - [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: N/A
   - [x] New dependencies: none
   - [x] `LICENSE`, `LICENSE-binary`, `NOTICE-binary`: N/A
   
   ### AI Tooling
   
   - [x] The PR includes the phrase "Contains content generated by Claude Code"
   - [x] My use of AI contributions follows the ASF legal policy https://www.apache.org/legal/generative-tooling.html
   
   Contains content generated by Claude Code.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)




> Replica with a corrupt meta header is not reported when a short-circuit read skips checksum verification
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-17986
>                 URL: https://issues.apache.org/jira/browse/HDFS-17986
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, hdfs-client
>    Affects Versions: 3.5.0, 3.4.3, 3.6.0
>            Reporter: Jose Luis López
>            Assignee: Jose Luis López
>            Priority: Major
>
> h3. Problem
> If a client reads with short-circuit reads enabled and checksum verification disabled (\{{setVerifyChecksum(false)}}), a replica with a corrupt meta file header is never reported to the NameNode:
> # The short-circuit client parses the meta header in \{{ShortCircuitReplica.<init>}}, gets a \{{CorruptMetaHeaderException}}, logs "error creating ShortCircuitReplica" in \{{BlockReaderFactory}}, and falls back to a remote read.
> # The remote read does not ask for checksums, so the DataNode never opens the meta file and never calls \{{handleBadBlock}} (HDFS-14706 only covers reads with checksums).
> # The read succeeds and nobody reports the replica. It is only found by the block scanner (up to 3 weeks by default) or by a later read with checksums.
> The failure is not cached, so every read of the block repeats the failed short-circuit attempt and falls back to TCP.
> h3. Who is affected
> HBase with its default \{{hbase.regionserver.checksum.verify=true}} and short-circuit reads. HBase reads the last block of each HFile with HDFS checksums when it opens the file, so corruption there is reported. Corruption in any other block of a multi-block HFile is not. HBASE-21607 is a production occurrence of the same fallback.
> Impact:
> * The NameNode counts a replica as healthy when nothing that verifies checksums (including re-replication) can use it.
> * The client loses short-circuit reads for that block.
> * If the data is also corrupt, HBase's own checksum usually catches it and retries with HDFS checksums, which triggers the DataNode report. HBASE-21607 shows this is not guaranteed: a failing disk zeroed the blocks, zeroed HFile blocks look like HBase's NULL checksum type, garbage was returned to the client, and the replica was never reported.
> h3. Reproduction
> HDFS 3.5.0 (1 DN, short-circuit enabled) with HBase 2.6.7 standalone, default settings:
> # Load 1M rows (\{{hbase pe ... sequentialWrite}}), then flush and major_compact. This gives one 256 MB HFile in 2 blocks.
> # Stop HBase, overwrite byte 2 (the checksum type) of the first block's \{{.meta}} file with 0x7f, and start HBase.
> # \{{count 'TestTable'}} returns all 1M rows. The RegionServer logs CorruptMetaHeaderException for the block on every scan, and \{{hdfs fsck}} stays HEALTHY.
> # Control: one \{{hdfs dfs -cat}} of the file (checksums on). The DataNode reports the block within seconds and fsck shows CORRUPT.
> Unit test \{{TestShortCircuitCorruptMetaHeader}} (needs native domain sockets) covers invalid and truncated headers with replication 1/3/5/7. On trunk, all cases pass with checksums on and all fail with checksums off.
> h3. Possible fix
> Have the DataNode check the meta header in \{{requestShortCircuitFdsForRead}} (FINALIZED, non-transient replicas only) and call \{{handleBadBlock}} if it is corrupt. Through the block scanner's suspect queue (HDFS-8224), that also verifies the data. It should still return the descriptors: failing the request makes the client disable short-circuit reads for the whole DataNode (see HDFS-12528).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org