You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jeff Zhang <zj...@gmail.com> on 2010/06/11 03:29:59 UTC
Is it safe enough to write edit log in buffer before writing to disk
?
Hi all,
I check the source code of EditLogFileOutputStream, it seems hadoop
will first write edit log to buffer, then flush to disk. I know that
it will improve performance, but in the other hand it will cause the
edit log in buffer lost when the name node is down. So I wonder is it
possible and necessary to let users do configuration in the conf file
to decide whether use buffer or not? Thanks.
--
Best Regards
Jeff Zhang
Re: Is it safe enough to write edit log in buffer before writing to
disk ?
Posted by Jeff Zhang <zj...@gmail.com>.
Thanks, Todd. I saw the flush code.
On Fri, Jun 11, 2010 at 9:31 AM, Todd Lipcon <to...@cloudera.com> wrote:
> Hi Jeff,
> All of the FSNamesystem methods call logSync() before returning to the
> client. So, if the edit is lost, it also will not have returned a success to
> the client.
> -Todd
>
> On Thu, Jun 10, 2010 at 6:29 PM, Jeff Zhang <zj...@gmail.com> wrote:
>>
>> Hi all,
>>
>> I check the source code of EditLogFileOutputStream, it seems hadoop
>> will first write edit log to buffer, then flush to disk. I know that
>> it will improve performance, but in the other hand it will cause the
>> edit log in buffer lost when the name node is down. So I wonder is it
>> possible and necessary to let users do configuration in the conf file
>> to decide whether use buffer or not? Thanks.
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Best Regards
Jeff Zhang
Re: Is it safe enough to write edit log in buffer before writing to
disk ?
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Jeff,
All of the FSNamesystem methods call logSync() before returning to the
client. So, if the edit is lost, it also will not have returned a success to
the client.
-Todd
On Thu, Jun 10, 2010 at 6:29 PM, Jeff Zhang <zj...@gmail.com> wrote:
> Hi all,
>
> I check the source code of EditLogFileOutputStream, it seems hadoop
> will first write edit log to buffer, then flush to disk. I know that
> it will improve performance, but in the other hand it will cause the
> edit log in buffer lost when the name node is down. So I wonder is it
> possible and necessary to let users do configuration in the conf file
> to decide whether use buffer or not? Thanks.
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Is it safe enough to write edit log in buffer before writing to
disk ?
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Jeff,
All of the FSNamesystem methods call logSync() before returning to the
client. So, if the edit is lost, it also will not have returned a success to
the client.
-Todd
On Thu, Jun 10, 2010 at 6:29 PM, Jeff Zhang <zj...@gmail.com> wrote:
> Hi all,
>
> I check the source code of EditLogFileOutputStream, it seems hadoop
> will first write edit log to buffer, then flush to disk. I know that
> it will improve performance, but in the other hand it will cause the
> edit log in buffer lost when the name node is down. So I wonder is it
> possible and necessary to let users do configuration in the conf file
> to decide whether use buffer or not? Thanks.
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
--
Todd Lipcon
Software Engineer, Cloudera