[jira] [Commented] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly

["Masatake Iwasaki (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940761#comment-14940761 ] 

Masatake Iwasaki commented on HDFS-1172:
----------------------------------------

bq. We'd better place the new "adding block to pending replica queue" logic only in checkReplication.

Thanks for the comment again. We can not get expected nodes in {{BlockManager#checkReplication}} because BlockUnderConstructionFeature is already removed by {{BlockInfo#convertToCompleteBlock}} at that point. I'm trying to update the pendingReplications only in the code path of {{completeFile}} now.

> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-1172
>                 URL: https://issues.apache.org/jira/browse/HDFS-1172
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Masatake Iwasaki
>         Attachments: HDFS-1172-150907.patch, HDFS-1172.008.patch, HDFS-1172.009.patch, HDFS-1172.010.patch, HDFS-1172.patch, hdfs-1172.txt, hdfs-1172.txt, replicateBlocksFUC.patch, replicateBlocksFUC1.patch, replicateBlocksFUC1.patch
>
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)