[jira] [Created] (HDFS-17984) Block recovery deletes the block when a restarting DataNode reports no replica before its replicas are loaded

["Bram Schuur (Jira)" <ji...@apache.org>]

Bram Schuur created HDFS-17984:
----------------------------------

             Summary: Block recovery deletes the block when a restarting DataNode reports no replica before its replicas are loaded
                 Key: HDFS-17984
                 URL: https://issues.apache.org/jira/browse/HDFS-17984
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode, hdfs
    Affects Versions: 3.5.0
         Environment: Hadoop hdfs 3.5.0
            Reporter: Bram Schuur


When a DataNode starts, {{DataNode#initBlockPool}} creates the FsDataset ({{{}initStorage{}}}) and only afterwards loads the block pool's replicas ({{{}FsDatasetImpl#addBlockPool{}}}), which first registers an empty replica map ({{{}volumeMap.initBlockPool{}}}) and then fills it ({{{}getAllVolumesMap{}}}). The IPC server is already running at that point. An {{initReplicaRecovery}} call from a block recovery primary in this window passes {{{}checkStorageState{}}}, finds no replica in the not yet filled map and returns null, i.e. "this DataNode does not have the block".

{{BlockRecoveryWorker#recover}} does not count a null answer as an error. If the other replicas fail, the sync list is empty and {{syncBlock}} commits with {{{}deleteBlock=true{}}}: the block is removed and the file is closed without it, although the restarting DataNode holds a valid replica, which it then deletes as an orphan once it has loaded.

Observed on 3.5.0, writing an HBase WAL (replication 2) during a rolling node restart. dn0 restarted at 11:22:43; dn1 (primary) ran the recovery at 11:22:46 and dn0 answered without an error:
{noformat}
11:22:43     [dn0] STARTUP_MSG: Starting DataNode
11:22:46,494 [dn1] java.io.IOException: Block length mismatch, len=7308711 but r=ReplicaBeingWritten, blk_1074458450_718705, RBW
11:22:46,494 [dn1] BlockRecoveryWorker: block=BP-...:blk_1074458450_718705 (length=0), isTruncateRecovery=false, syncList=[]
11:22:46,495 [nn]  commitBlockSynchronization(oldBlock=BP-...:blk_1074458450_718705, newgenerationstamp=718982, newlength=0, newtargets=[], closeFile=true, deleteBlock=true)
11:22:55,864 [dn0] Scheduling blk_1074458450_718705 replica ReplicaWaitingToBeRecovered, blk_1074458450_718705, RWR ... for deletion
{noformat}
A few hundred milliseconds earlier (11:22:46.19) dn0 answered another recovery with "Storage not yet initialized for initReplicaRecovery"; that case is already an error. The gap is between {{initStorage}} and the end of {{{}addBlockPool{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org