You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Jose Luis López (Jira)" <ji...@apache.org> on 2026/10/03 21:07:00 UTC
[jira] [Created] (HDFS-17986) Replica with a corrupt meta header is not reported when a short-circuit read skips checksum verification
Jose Luis López created HDFS-17986:
--------------------------------------
Summary: Replica with a corrupt meta header is not reported when a short-circuit read skips checksum verification
Key: HDFS-17986
URL: https://issues.apache.org/jira/browse/HDFS-17986
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, hdfs-client
Affects Versions: 3.4.3, 3.5.0, 3.6.0
Reporter: Jose Luis López
Assignee: Jose Luis López
h3. Problem
If a client reads with short-circuit reads enabled and checksum verification disabled (\{{setVerifyChecksum(false)}}), a replica with a corrupt meta file header is never reported to the NameNode:
# The short-circuit client parses the meta header in \{{ShortCircuitReplica.<init>}}, gets a \{{CorruptMetaHeaderException}}, logs "error creating ShortCircuitReplica" in \{{BlockReaderFactory}}, and falls back to a remote read.
# The remote read does not ask for checksums, so the DataNode never opens the meta file and never calls \{{handleBadBlock}} (HDFS-14706 only covers reads with checksums).
# The read succeeds and nobody reports the replica. It is only found by the block scanner (up to 3 weeks by default) or by a later read with checksums.
The failure is not cached, so every read of the block repeats the failed short-circuit attempt and falls back to TCP.
h3. Who is affected
HBase with its default \{{hbase.regionserver.checksum.verify=true}} and short-circuit reads. HBase reads the last block of each HFile with HDFS checksums when it opens the file, so corruption there is reported. Corruption in any other block of a multi-block HFile is not. HBASE-21607 is a production occurrence of the same fallback.
Impact:
* The NameNode counts a replica as healthy when nothing that verifies checksums (including re-replication) can use it.
* The client loses short-circuit reads for that block.
* If the data is also corrupt, HBase's own checksum usually catches it and retries with HDFS checksums, which triggers the DataNode report. HBASE-21607 shows this is not guaranteed: a failing disk zeroed the blocks, zeroed HFile blocks look like HBase's NULL checksum type, garbage was returned to the client, and the replica was never reported.
h3. Reproduction
HDFS 3.5.0 (1 DN, short-circuit enabled) with HBase 2.6.7 standalone, default settings:
# Load 1M rows (\{{hbase pe ... sequentialWrite}}), then flush and major_compact. This gives one 256 MB HFile in 2 blocks.
# Stop HBase, overwrite byte 2 (the checksum type) of the first block's \{{.meta}} file with 0x7f, and start HBase.
# \{{count 'TestTable'}} returns all 1M rows. The RegionServer logs CorruptMetaHeaderException for the block on every scan, and \{{hdfs fsck}} stays HEALTHY.
# Control: one \{{hdfs dfs -cat}} of the file (checksums on). The DataNode reports the block within seconds and fsck shows CORRUPT.
Unit test \{{TestShortCircuitCorruptMetaHeader}} (needs native domain sockets) covers invalid and truncated headers with replication 1/3/5/7. On trunk, all cases pass with checksums on and all fail with checksums off.
h3. Possible fix
Have the DataNode check the meta header in \{{requestShortCircuitFdsForRead}} (FINALIZED, non-transient replicas only) and call \{{handleBadBlock}} if it is corrupt. Through the block scanner's suspect queue (HDFS-8224), that also verifies the data. It should still return the descriptors: failing the request makes the client disable short-circuit reads for the whole DataNode (see HDFS-12528).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org