You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Kihwal Lee (JIRA)" <ji...@apache.org> on 2014/01/30 22:34:19 UTC

[jira] [Commented] (HDFS-4183) Throttle block recovery

    [ https://issues.apache.org/jira/browse/HDFS-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13887091#comment-13887091 ] 

Kihwal Lee commented on HDFS-4183:
----------------------------------

I think the overhead of block recoveries caused by massive lease expiration will drastically reduce after HDFS-5790. 

> Throttle block recovery
> -----------------------
>
>                 Key: HDFS-4183
>                 URL: https://issues.apache.org/jira/browse/HDFS-4183
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 0.23.4, 2.0.2-alpha
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>
> When a large number of files are abandoned without closing, a storm of lease expiration follows in about an hour (lease hard limit). For the last block of each file, block recovery is initiated and when the datanode is done, it calls commitBlockSynchronization() is called against namenode. A burst of these calls can slow down namenode considerably. We need to throttle block recovery and/or speed up the rate at which commitBlockSynchronization() is served.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)