[jira] [Commented] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly

["Ravi Prakash (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675139#comment-13675139 ] 

Ravi Prakash commented on HDFS-1172:
------------------------------------

I am able to consistently reproduce this issue with the following command on an 80 node cluster:
hadoop jar $HADOOP_PREFIX/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar SliveTest -baseDir /user/someUser/slive -duration 120 -dirSize 122500 -files 122500 -maps 560 -reduces 1 -seed 1 -ops 100 -readSize 1048576,1048576 -writeSize 1048576,1048576 -appendSize 1048576,1048576 -replication 1,1 -blockSize 1024,1024 -delete 0,uniform -create 100,uniform -mkdir 0,uniform -rename 0,uniform -append 0,uniform -ls 0,uniform -read 0,uniform

This litters the task logs with the NotReplicatedYetException
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1268)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:469)


                
> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-1172
>                 URL: https://issues.apache.org/jira/browse/HDFS-1172
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>             Fix For: 0.24.0
>
>         Attachments: HDFS-1172.patch, hdfs-1172.txt, hdfs-1172.txt, replicateBlocksFUC1.patch, replicateBlocksFUC1.patch, replicateBlocksFUC.patch
>
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira