You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Ravi Prakash (JIRA)" <ji...@apache.org> on 2011/09/08 00:05:10 UTC
[jira] [Commented] (HDFS-1172) Blocks in newly completed files are
considered under-replicated too quickly
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099604#comment-13099604 ]
Ravi Prakash commented on HDFS-1172:
------------------------------------
Thanks Todd for taking care of this! :)
+1 to the patch
- Nitpicking, should we just have a boolean cached for the isLastBlockOfUnderConstructionFile calls on 1131 and 1063?
- Is line 1225 {noformat}return lastBlock == block;{noformat} the same as an equality check?
- In TestReplication.java:testReplicationWhileUnderConstruction(), after marking one block as bad (line 588), is there a quick check we can do to verify that indeed a block was added to the pending queue?
> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
> Key: HDFS-1172
> URL: https://issues.apache.org/jira/browse/HDFS-1172
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.21.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: HDFS-1172.patch, hdfs-1172.txt, replicateBlocksFUC.patch, replicateBlocksFUC1.patch, replicateBlocksFUC1.patch
>
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira