You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "cxzl25 (via GitHub)" <gi...@apache.org> on 2026/04/08 13:55:56 UTC

Re: [PR] HDFS-16942. Send error to datanode if FBR is rejected due to bad lease [hadoop]

cxzl25 commented on code in PR #5460:
URL: https://github.com/apache/hadoop/pull/5460#discussion_r3051817271


##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java:
##########
@@ -791,6 +792,9 @@ private void offerService() throws Exception {
           shouldServiceRun = false;
           return;
         }
+        if (InvalidBlockReportLeaseException.class.getName().equals(reClass)) {
+          fullBlockReportLeaseId = 0;

Review Comment:
   ## Description
   
   HDFS-16942 introduced `InvalidBlockReportLeaseException`, which the NameNode now throws back to the DataNode via RPC when a block report is rejected due to an invalid lease. On a DataNode that also includes HDFS-16942, the exception is caught and `fullBlockReportLeaseId` is reset to 0, allowing the DN to request a new lease on the next heartbeat and retry.
   
   However, during a rolling upgrade where the NameNode has been upgraded (with HDFS-16942) but DataNodes are still running an older version (without HDFS-16942), the old DataNode code does not have the `InvalidBlockReportLeaseException` handling branch in `BPServiceActor.offerService()`. This causes the DN to enter an infinite failure loop where it can never successfully send a full block report.
   
   ## Root Cause
   
   In `BPServiceActor.offerService()`, the logic works as follows:
   
   1. The DN requests a block report lease during heartbeat only when `fullBlockReportLeaseId == 0`:
   
   ```java
   boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
           scheduler.isBlockReportDue(startTime);
   ```
   
   2. After sending the block report, `fullBlockReportLeaseId` is reset to 0:
   
   ```java
   if ((fullBlockReportLeaseId != 0) || forceFullBr) {
       cmds = blockReport(fullBlockReportLeaseId);
       fullBlockReportLeaseId = 0;  // not reached if blockReport() throws
   }
   ```
   
   3. When the upgraded NN throws `InvalidBlockReportLeaseException`, `blockReport()` propagates the exception. The `fullBlockReportLeaseId = 0` line after the call is never executed.
   
   4. The exception is caught by the generic `RemoteException` catch block. The old DN code does not recognize `InvalidBlockReportLeaseException`, so `fullBlockReportLeaseId` remains set to the stale invalid value.
   
   5. On the next heartbeat iteration, because `fullBlockReportLeaseId != 0`, `requestBlockReportLease` is `false` — the DN does not request a new lease. It then attempts `blockReport()` again with the same stale lease, which the NN rejects again. This repeats indefinitely.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org