[jira] [Commented] (HDFS-744) Support hsync in HDFS

["Luke Lu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462578#comment-13462578 ] 

Luke Lu commented on HDFS-744:
------------------------------

I think that it's important enough to clarify:

bq. When hsync returns the DNs have all gotten the data and the request to flush and sync, but they are not necessarily finished with it.

It's NOT the "proper" fsync semantics then. In fact, this is even worse than hflush in branch-1, which acks after data is flushed to OS cache (written to FileOutputStream). i.e., HBase with Hadoop 1.x can survive kill -9 all DNs at the same time without data loss, while this hsync implementation cannot. 

bq. The same, BTW, is currently (without my patch) true to hflush. When hflush returns the DNs have received the data in their buffers, but they may not be done flushing the data to the OS buffers.

This is an unfortunate side-effect of HDS-265 that makes branch-1 hflush more durable (which is not enough) than hsync/hflush in trunk (and all post 0.21 code).

bq. If we wanted to fix this (which would involve doing the flushOrSync in the PacketResponder if the request is a client request, which inconveniently would also serialize the flush/syncs across replicas), this should be a different jira, I think.

Did you file a JIRA Lars? BTW, instead of doing the sync in the responder (ack thread), an alternative implementation could use a conditional variable to signal sync done in the data thread..., which I think, is a small price to pay for a parallel sync with the correct hsync semantics.
                
> Support hsync in HDFS
> ---------------------
>
>                 Key: HDFS-744
>                 URL: https://issues.apache.org/jira/browse/HDFS-744
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Hairong Kuang
>            Assignee: Lars Hofhansl
>             Fix For: 2.0.2-alpha
>
>         Attachments: HDFS-744-2.0-v1.patch, HDFS-744-2.0-v2.patch, HDFS-744-trunk.patch, HDFS-744-trunk-v2.patch, HDFS-744-trunk-v3.patch, HDFS-744-trunk-v4.patch, HDFS-744-trunk-v5.patch, HDFS-744-trunk-v6.patch, HDFS-744-trunk-v7.patch, HDFS-744-trunk-v8.patch, hdfs-744.txt, hdfs-744-v2.txt, hdfs-744-v3.txt
>
>
> HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, the real expected semantics should be "flushes out to all replicas and all replicas have done posix fsync equivalent - ie the OS has flushed it to the disk device (but the disk may have it in its cache)." This jira aims to implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira