You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hawq.apache.org by Paul Guo <pa...@gmail.com> on 2016/12/12 07:56:00 UTC

sync related issues in hdfs IO

1) In MirroredAppendOnly_FlushAndClose()

    {

        int     ret;


        errno = 0;


        ret = FileSync(open->primaryFile);

        if (ret != 0)

            *primaryError = errno;

    }


    FileClose(open->primaryFile);



If I understand the code correctly, MirroredAppendOnly_DoOpen() sets

O_SYNC in file flags which leads to a hsync in FileClose(). So it looks

that FileSync() is not needed here to protect the data integrity.


2)

 /*

 * flush hdfs file

 *

 * NB: hdfs flush do NOT promise that data has been written on disk

 * after flush, data can be read by others

 */

 int

 HdfsFileSync(File file)

It seems that HdfsFileSync finally does hsync finally, so I assume this
comment is wrong, right?

Re: sync related issues in hdfs IO

Posted by Ma Hongxu <in...@outlook.com>.

Hi paul

Question 2, I agreed with you.

Or the comment means: "since flush do not promise data has been written,  so we need sync here to promise it..."


Overall it do sync rather than flush here.


在 12/12/2016 15:56, Paul Guo 写道:

1) In MirroredAppendOnly_FlushAndClose()

    {

        int     ret;


        errno = 0;


        ret = FileSync(open->primaryFile);

        if (ret != 0)

            *primaryError = errno;

    }


    FileClose(open->primaryFile);



If I understand the code correctly, MirroredAppendOnly_DoOpen() sets

O_SYNC in file flags which leads to a hsync in FileClose(). So it looks

that FileSync() is not needed here to protect the data integrity.


2)

 /*

 * flush hdfs file

 *

 * NB: hdfs flush do NOT promise that data has been written on disk

 * after flush, data can be read by others

 */

 int

 HdfsFileSync(File file)

It seems that HdfsFileSync finally does hsync finally, so I assume this
comment is wrong, right?