[jira] [Created] (HDFS-7980) Incremental BlockReport will dramatically slow down the startup of a namenode

["Hui Zheng (JIRA)" <ji...@apache.org>]

Hui Zheng created HDFS-7980:
-------------------------------

             Summary: Incremental BlockReport will dramatically slow down the startup of  a namenode
                 Key: HDFS-7980
                 URL: https://issues.apache.org/jira/browse/HDFS-7980
             Project: Hadoop HDFS
          Issue Type: Bug
            Reporter: Hui Zheng


In the current implementation the datanode will call the reportReceivedDeletedBlocks() method that is a IncrementalBlockReport before calling the bpNamenode.blockReport() method. So in a large(several thousands of datanodes) and busy cluster it will slow down(more than one hour) the startup of namenode. 

{code}
List<DatanodeCommand> blockReport() throws IOException {
    // send block report if timer has expired.
    final long startTime = now();
    if (startTime - lastBlockReport <= dnConf.blockReportInterval) {
      return null;
    }

    final ArrayList<DatanodeCommand> cmds = new ArrayList<DatanodeCommand>();

    // Flush any block information that precedes the block report. Otherwise
    // we have a chance that we will miss the delHint information
    // or we will report an RBW replica after the BlockReport already reports
    // a FINALIZED one.
    reportReceivedDeletedBlocks();
    lastDeletedReport = startTime;
    .........
        // Send the reports to the NN.
    int numReportsSent = 0;
    int numRPCs = 0;
    boolean success = false;
    long brSendStartTime = now();
    try {
      if (totalBlockCount < dnConf.blockReportSplitThreshold) {
        // Below split threshold, send all reports in a single message.
        DatanodeCommand cmd = bpNamenode.blockReport(
            bpRegistration, bpos.getBlockPoolId(), reports);
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)