[jira] [Commented] (HDFS-6489) DFS Used space is not correct computed on frequent append operations

["Yongjun Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-6489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16480265#comment-16480265 ] 

Yongjun Zhang commented on HDFS-6489:
-------------------------------------

HI Guys,

Thanks for working on this issue. I have gone through the discussions, would like to share my thoughts:

- The append operation tend to happen many times on the same block, this makes it easy to deny write due to incorrect DU estimation of append.
- [~cheersyang]'s approach tries to remedy the situation by "interrupts DURefreshThread and then evaluates the space again", however, it would be too slow to help ongoing writes.
- [~raviprak]'s approach tries not to include block size when incr DU when converting complete block to RBW (append). This might under estimate the disk usage, and cause over subscription of a DN capacity, and cause write to fail. The fs.du.interval is 10 minutes by default, the oversubscription (if any) can be corrected in 10 minutes (right Ravi?). So the chance of this failure is relatively low. 

Sounds like we can go with Ravi's solution if the accounting is corrected every 10 minutes. Given that we don't have a perfect solution for this problem, and DU is an estimation anyways.

What do you guys think?

Thanks.










> DFS Used space is not correct computed on frequent append operations
> --------------------------------------------------------------------
>
>                 Key: HDFS-6489
>                 URL: https://issues.apache.org/jira/browse/HDFS-6489
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.2.0, 2.7.1, 2.7.2
>            Reporter: stanley shi
>            Priority: Major
>         Attachments: HDFS-6489.001.patch, HDFS-6489.002.patch, HDFS-6489.003.patch, HDFS-6489.004.patch, HDFS-6489.005.patch, HDFS-6489.006.patch, HDFS-6489.007.patch, HDFS6489.java
>
>
> The current implementation of the Datanode will increase the DFS used space on each block write operation. This is correct in most scenario (create new file), but sometimes it will behave in-correct(append small data to a large block).
> For example, I have a file with only one block(say, 60M). Then I try to append to it very frequently but each time I append only 10 bytes;
> Then on each append, dfs used will be increased with the length of the block(60M), not teh actual data length(10bytes).
> Consider in a scenario I use many clients to append concurrently to a large number of files (1000+), assume the block size is 32M (half of the default value), then the dfs used will be increased 1000*32M = 32G on each append to the files; but actually I only write 10K bytes; this will cause the datanode to report in-sufficient disk space on data write.
> {quote}2014-06-04 15:27:34,719 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock  BP-1649188734-10.37.7.142-1398844098971:blk_1073742834_45306 received exception org.apache.hadoop.util.DiskChecker$DiskOutOfSpaceException: Insufficient space for appending to FinalizedReplica, blk_1073742834_45306, FINALIZED{quote}
> But the actual disk usage:
> {quote}
> [root@hdsh143 ~]# df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda3              16G  2.9G   13G  20% /
> tmpfs                 1.9G   72K  1.9G   1% /dev/shm
> /dev/sda1              97M   32M   61M  35% /boot
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org