[jira] Commented: (HADOOP-2656) Support for upgrading existing cluster to facilitate appends to HDFS files

["dhruba borthakur (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576454#action_12576454 ] 

dhruba borthakur commented on HADOOP-2656:
------------------------------------------

The Datanode needs to store a block generation stamp for each block. The original idea was to store the block generation stamp inside the meta file of each block. One major disadvantage of this approach is that generation of a block report requires that each meta file be opened and the generation stamp read from it. With about 50K blocks per datanode and a seek time of 10 ms, this might require 300 seconds. This means that the time to restart a cluster cannot be lesser than this 300 seconds.

A few other alternatives:
1. Encode the generation stamp into the name of the metafile. Each metafile will look like blkxxxxxx.genstamp.meta. The block file will remain the same.

2. Encode the generation stamp into the name of the block file. Each block file will be of the form blkxxxxxx.genstamp. The metafile will remain the same.

3. Encode the generation stamp into the name of a new zero-size file named blkxxxxx.genstamp. The block file and the metadata file will remain the same.

4. A completely separate file (one per datanode) that records the metadata of all blocks in the datanode.

I propose that we implement option 1.

> Support for upgrading existing cluster to facilitate appends to HDFS files
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-2656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2656
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>
> HADOOP-1700 describes the design for supporting appends to HDFS files. This design requires a distributed-upgrade to existing cluster installations. The design specifies that the DataNode persist the 8-byte BlockGenerationStamp in the block metadata file. The upgrade code will introduce this new field in the block metadata file and initialize this value to 0.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.