[jira] [Created] (HDFS-17969) Block missing due to re-registration race after failover in rolling upgrade

["lei w (Jira)" <ji...@apache.org>]

lei w created HDFS-17969:
----------------------------

             Summary: Block missing due to re-registration race after failover in rolling upgrade
                 Key: HDFS-17969
                 URL: https://issues.apache.org/jira/browse/HDFS-17969
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode
            Reporter: lei w


Our cluster upgrade follows this procedure:
1. The active NameNode is switched to standby (failover).
2. The (now standby) node is upgraded while the other node serves as active.
3. After the upgrade completes, the upgraded node is switched back to active.
Block missing was observed under the following race condition, which occurs when the upgraded node is switched back to active.

Root cause analysis:

1. NN restart/failover completes; the in-memory datanodemap is reset.
2. DN Actor thread sends a heartbeat; since the NN no longer recognizes this DN, the heartbeat response carries a registration command. Every DN receives this registration command after the failover.
3. The Actor thread enqueues the command into the command queue, which is consumed by the cmdThread.
4. The cmdThread starts registration. Since the NN is busy at this point (processing block reports from all DNs), the retrieveNamespaceInfo call preceding registration takes a long time and may even time out.
5. Because step 4 can take longer than the heartbeat interval, the Actor thread sends the next heartbeat before registration finishes. Upon success, the NN returns another registration command, which is enqueued again.
6. The first registration succeeds; the cmdThread schedules a full block report.
7. The Actor thread notices the scheduled full block report and triggers the BRThread.
8. The BRThread performs the full block report (covering both incremental and stored blocks). This takes a long time.
9. Meanwhile, the cmdThread picks up the second registration command from the queue and starts registration again (retrieveNamespaceInfo, clear ibr, register). After it completes, another full block report is scheduled.
10. The BRThread finishes the full block report and reschedules the next full block report to a random time within the configured FBR window .
Any blocks written between the moment the incremental block report (IBR) is sent in step 8 and the moment clear ibr executes in step 9 will not be reported to the NN until the next full block report.  If all DataNodes holding replicas of a given block are in this state, the NN never receives any report for that block within the gap window.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org