[jira] [Commented] (HDFS-17984) Block recovery deletes the block when a restarting DataNode reports no replica before its replicas are loaded

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-17984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18121906#comment-18121906 ] 

ASF GitHub Bot commented on HDFS-17984:
---------------------------------------

craffit opened a new pull request, #8778:
URL: https://github.com/apache/hadoop/pull/8778

   ### Description of PR
   
   JIRA: https://issues.apache.org/jira/browse/HDFS-17984
   
   While a DataNode is starting, `initReplicaRecovery` can be served after the FsDataset exists but before the block pool's replicas are loaded into the replica map. It then answers "no replica" (null), which `BlockRecoveryWorker` does not count as an error; if the other replicas fail, the sync list is empty and the block is deleted although the restarting DataNode holds a valid replica.
   
   `FsDatasetImpl` now records per block pool when its replicas have been loaded (`addBlockPool`, cleared in `shutdownBlockPool`), and `initReplicaRecovery` throws while they are not, so the DataNode counts as failed and block recovery is retried. A no-op `DataNodeFaultInjector#delayLoadingReplicas` hook is added before the replicas are loaded.
   
   `TestInterDatanodeProtocol` used a non-existent block pool to get a "missing block" null response. A DataNode cannot tell such a pool from one it has not loaded yet, so the test now checks a missing block in the loaded pool for null, and that an unknown pool is rejected.
   
   ### How was this patch tested?
   
   Added `TestBlockRecovery2#testBlockRecoveryWhileReplicasAreLoading`: one replica is unusable and the other DataNode is held while loading its replicas after a restart; it checks that the file is recovered with the restarting DataNode's replica instead of being closed without the block.
   
   ### For code changes:
   
   - [x] Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
   - [ ] Object storage: Have the integration tests been executed and the endpoint declared according to the connector-specific documentation? (N/A)
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? (N/A)
   - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? (N/A)
   
   ### AI Tooling
   
   If an AI tool was used:
   
   - [x] The PR includes the phrase "Contains content generated by <tool>" where <tool> is the name of the AI tool used.
   - [x] My use of AI contributions follows the ASF legal policy https://www.apache.org/legal/generative-tooling.html
   
   Contains content generated by Claude Code.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   




> Block recovery deletes the block when a restarting DataNode reports no replica before its replicas are loaded
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-17984
>                 URL: https://issues.apache.org/jira/browse/HDFS-17984
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode, hdfs
>    Affects Versions: 3.5.0
>         Environment: Hadoop hdfs 3.5.0
>            Reporter: Bram Schuur
>            Priority: Major
>
> When a DataNode starts, {{DataNode#initBlockPool}} creates the FsDataset ({{{}initStorage{}}}) and only afterwards loads the block pool's replicas ({{{}FsDatasetImpl#addBlockPool{}}}), which first registers an empty replica map ({{{}volumeMap.initBlockPool{}}}) and then fills it ({{{}getAllVolumesMap{}}}). The IPC server is already running at that point. An {{initReplicaRecovery}} call from a block recovery primary in this window passes {{{}checkStorageState{}}}, finds no replica in the not yet filled map and returns null, i.e. "this DataNode does not have the block".
> {{BlockRecoveryWorker#recover}} does not count a null answer as an error. If the other replicas fail, the sync list is empty and {{syncBlock}} commits with {{{}deleteBlock=true{}}}: the block is removed and the file is closed without it, although the restarting DataNode holds a valid replica, which it then deletes as an orphan once it has loaded.
> Observed on 3.5.0, writing an HBase WAL (replication 2) during a rolling node restart. dn0 restarted at 11:22:43; dn1 (primary) ran the recovery at 11:22:46 and dn0 answered without an error:
> {noformat}
> 11:22:43     [dn0] STARTUP_MSG: Starting DataNode
> 11:22:46,494 [dn1] java.io.IOException: Block length mismatch, len=7308711 but r=ReplicaBeingWritten, blk_1074458450_718705, RBW
> 11:22:46,494 [dn1] BlockRecoveryWorker: block=BP-...:blk_1074458450_718705 (length=0), isTruncateRecovery=false, syncList=[]
> 11:22:46,495 [nn]  commitBlockSynchronization(oldBlock=BP-...:blk_1074458450_718705, newgenerationstamp=718982, newlength=0, newtargets=[], closeFile=true, deleteBlock=true)
> 11:22:55,864 [dn0] Scheduling blk_1074458450_718705 replica ReplicaWaitingToBeRecovered, blk_1074458450_718705, RWR ... for deletion
> {noformat}
> A few hundred milliseconds earlier (11:22:46.19) dn0 answered another recovery with "Storage not yet initialized for initReplicaRecovery"; that case is already an error. The gap is between {{initStorage}} and the end of {{{}addBlockPool{}}}.
> Related: 
> https://issues.apache.org/jira/browse/HDFS-16097
> https://issues.apache.org/jira/browse/HDFS-16717



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org