HDFS: fault tolerance to block losses with namenode failure

[Cagdas Gerede <ca...@gmail.com>]

If you are writing 10 blocks for a file and let's say in 10th block namenode
fails, all previous 9 blocks are lost because you were not able to close the
file and therefore namenode did not persist the information about 9 blocks
to the fsimage file.

How would you solve this problem in the application? Why does the hdfs
client make namenode persist every block once the block is written and
instead waits until closing of the file? Then, don't you need to keep a copy
of all the blocks in your application until you close the file successfully
to prevent data loss. Does it make sense to have this semantics with the
assumption of very large files with multiple blocks?

Thanks for your response,

-- 
------------
Best Regards, Cagdas Evren Gerede
Home Page: http://cagdasgerede.info

Re: HDFS: fault tolerance to block losses with namenode failure

[Dhruba Borthakur <dh...@gmail.com>]

Starting in 0.17 release, an application can invoke
DFSOutputStream.fsync() to persist block locations for a file even
before the file is closed.

thanks,
dhruba


On Tue, May 6, 2008 at 8:11 AM, Cagdas Gerede <ca...@gmail.com> wrote:
> If you are writing 10 blocks for a file and let's say in 10th block namenode
>  fails, all previous 9 blocks are lost because you were not able to close the
>  file and therefore namenode did not persist the information about 9 blocks
>  to the fsimage file.
>
>  How would you solve this problem in the application? Why does the hdfs
>  client make namenode persist every block once the block is written and
>  instead waits until closing of the file? Then, don't you need to keep a copy
>  of all the blocks in your application until you close the file successfully
>  to prevent data loss. Does it make sense to have this semantics with the
>  assumption of very large files with multiple blocks?
>
>  Thanks for your response,
>
>  --
>  ------------
>  Best Regards, Cagdas Evren Gerede
>  Home Page: http://cagdasgerede.info
>