[jira] [Created] (HDFS-17983) Lease recovery rejects an RBW replica with "Block length mismatch" after an interrupted packet write

["Bram Schuur (Jira)" <ji...@apache.org>]

Bram Schuur created HDFS-17983:
----------------------------------

             Summary:  Lease recovery rejects an RBW replica with "Block length mismatch" after an interrupted packet write
                 Key: HDFS-17983
                 URL: https://issues.apache.org/jira/browse/HDFS-17983
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode, hdfs
    Affects Versions: 3.5.0
         Environment: Hadoop hdfs version 3.5.0
            Reporter: Bram Schuur


When a packet write on a DataNode fails after the data has reached the block file but before bytesOnDisk is updated, the RBW replica is left with a block file longer than its in-memory bytesOnDisk. One way this happens: *BlockReceiver#receivePacket* writes the data and checksum, the PacketResponder interrupts the receiver because the client went away, and the following fsync throws *ClosedByInterruptException*. The extra bytes were never acknowledged.

Pipeline recovery has handled this case since HDFS-11472: *FsDatasetImpl#recoverRbwImpl* resets bytesOnDisk and truncates to the acknowledged length. Lease/block recovery does not. *FsDatasetImpl#initReplicaRecoveryImpl* calls *checkReplicaFiles*, which requires *bytesOnDisk == blockDataLength* and throws otherwise. The replica is excluded from recovery even though it holds all acknowledged data:
  - if it is the only replica, recovery keeps failing ("All datanodes failed") until the DataNode restarts and reloads the replica as RWR with a checksum-validated length;
  - if the other replicas cannot be used either, the block can be lost.

Observed on 3.5.0, writing an HBase WAL (hsync, replication 2) during a rolling node restart:
{code}
 11:21:40,998 INFO datanode.DataNode: opWriteBlock BP-...:blk_1074458450_718705 received exception java.nio.channels.ClosedByInterruptException
 11:22:46,494 INFO impl.FsDatasetImpl: initReplicaRecovery: blk_1074458450_718705, recoveryId=718982, replica=ReplicaBeingWritten, blk_1074458450_718705, RBW
 11:22:46,494 WARN protocol.InterDatanodeProtocol: Failed to recover block (block=BP-...:blk_1074458450_718705, datanode=DatanodeInfoWithStorage[dn1:50010,null,null])
 java.io.IOException: Block length mismatch, len=7308711 but r=ReplicaBeingWritten, blk_1074458450_718705, RBW
 11:22:46,494 INFO datanode.DataNode: BlockRecoveryWorker: block=BP-...:blk_1074458450_718705 (length=0), isTruncateRecovery=false, syncList=[]
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org