You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/10/02 08:59:00 UTC

[jira] [Commented] (HDFS-17983) Lease recovery rejects an RBW replica with "Block length mismatch" after an interrupted packet write

    [ https://issues.apache.org/jira/browse/HDFS-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121886#comment-18121886 ] 

ASF GitHub Bot commented on HDFS-17983:
---------------------------------------

craffit opened a new pull request, #8777:
URL: https://github.com/apache/hadoop/pull/8777

   ### Description of PR
   
   JIRA: https://issues.apache.org/jira/browse/HDFS-17983
   
   A packet write that fails after its data reached the block file but before `bytesOnDisk` is updated (e.g. `ClosedByInterruptException` while syncing) leaves an RBW replica whose block file is longer than `bytesOnDisk`. Lease/block recovery then rejects that replica in `checkReplicaFiles` ("Block length mismatch"), even though it holds all acknowledged data.
   
   `FsDatasetImpl#initReplicaRecoveryImpl` now truncates the block and meta file to `bytesOnDisk` in that case, as `recoverRbwImpl` already does for pipeline recovery since HDFS-11472. The existing `bytesOnDisk >= visibleLength` check runs first, so no acknowledged bytes are dropped.
   
   ### How was this patch tested?
   
   Added `TestLeaseRecovery#testLeaseRecoveryAfterInterruptedPacketWrite`, which interrupts the DataNode receiver after a packet's data is written so the following fsync fails, and checks that lease recovery closes the file at the acknowledged length.
   
   ### For code changes:
   
   - [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: Have the integration tests been executed and the endpoint declared according to the connector-specific documentation? (N/A)
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? (N/A)
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? (N/A)
   
   ### AI Tooling
   
   If an AI tool was used:
   
   - [x] The PR includes the phrase "Contains content generated by <tool>" where <tool> is the name of the AI tool used.
   - [x] My use of AI contributions follows the ASF legal policy https://www.apache.org/legal/generative-tooling.html
   
   Contains content generated by Claude Code.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)




>  Lease recovery rejects an RBW replica with "Block length mismatch" after an interrupted packet write
> -----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-17983
>                 URL: https://issues.apache.org/jira/browse/HDFS-17983
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode, hdfs
>    Affects Versions: 3.5.0
>         Environment: Hadoop hdfs version 3.5.0
>            Reporter: Bram Schuur
>            Priority: Major
>
> When a packet write on a DataNode fails after the data has reached the block file but before bytesOnDisk is updated, the RBW replica is left with a block file longer than its in-memory bytesOnDisk. One way this happens: *BlockReceiver#receivePacket* writes the data and checksum, the PacketResponder interrupts the receiver because the client went away, and the following fsync throws *ClosedByInterruptException*. The extra bytes were never acknowledged.
> Pipeline recovery has handled this case since HDFS-11472: *FsDatasetImpl#recoverRbwImpl* resets bytesOnDisk and truncates to the acknowledged length. Lease/block recovery does not. *FsDatasetImpl#initReplicaRecoveryImpl* calls *checkReplicaFiles*, which requires *bytesOnDisk == blockDataLength* and throws otherwise. The replica is excluded from recovery even though it holds all acknowledged data:
>   - if it is the only replica, recovery keeps failing ("All datanodes failed") until the DataNode restarts and reloads the replica as RWR with a checksum-validated length;
>   - if the other replicas cannot be used either, the block can be lost.
> Observed on 3.5.0, writing an HBase WAL (hsync, replication 2) during a rolling node restart:
> {code}
>  11:21:40,998 INFO datanode.DataNode: opWriteBlock BP-...:blk_1074458450_718705 received exception java.nio.channels.ClosedByInterruptException
>  11:22:46,494 INFO impl.FsDatasetImpl: initReplicaRecovery: blk_1074458450_718705, recoveryId=718982, replica=ReplicaBeingWritten, blk_1074458450_718705, RBW
>  11:22:46,494 WARN protocol.InterDatanodeProtocol: Failed to recover block (block=BP-...:blk_1074458450_718705, datanode=DatanodeInfoWithStorage[dn1:50010,null,null])
>  java.io.IOException: Block length mismatch, len=7308711 but r=ReplicaBeingWritten, blk_1074458450_718705, RBW
>  11:22:46,494 INFO datanode.DataNode: BlockRecoveryWorker: block=BP-...:blk_1074458450_718705 (length=0), isTruncateRecovery=false, syncList=[]
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org