[jira] [Commented] (HDFS-3960) Snapshot of Being Written Files

["Todd Lipcon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-3960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13459760#comment-13459760 ] 

Todd Lipcon commented on HDFS-3960:
-----------------------------------

Off-the-cuff brainstormy idea:

1. When the client calls hflush() or hsync(), set a flag on the packet which indicates to the DN that the block is an "durable" (has been hflushed at least once) block. When the client calls persistBlocks() to the NN to indicate an fsynced block, the NN similarly sets a flag.
2. On heartbeats, the DN reports to the DN the visible length of any "durable" blocks which are open-for-write
3. The NN generally ignores this info on the heartbeat. However, when a user requests a snapshot, he or she must wait until every open-for-write durable block has been reported by a DN. This process "fills in" the length of the blocks

Alternatively, instead of 3, the NN could always update the block length for these blocks on heartbeat, which would make the "take a snapshot" operation non-blocking, but might have some impact on NN load.

My hunch is that, in the average HDFS cluster, the number of files which are being written "durably" (ie fsynced often), is very low - probably in the thousands at most. So the extra impact NN load seems workable.




                
> Snapshot of Being Written Files
> -------------------------------
>
>                 Key: HDFS-3960
>                 URL: https://issues.apache.org/jira/browse/HDFS-3960
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Tsz Wo (Nicholas), SZE
>
> Here is a design question: Suppose there is a being written file when a snapshot is being taken.  What should the length of the file be shown in the snapshot?  In other words, how to determine the length of being written file when a snapshot is being taken?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira