HLog and durability question --0.90 and 0.20

[Sean Bigdatafun <se...@gmail.com>]

Can someone give me a detailed look at the HLog mechanism for 0.90
durablity?
I recall that HBase committers claim that data will be truly durable in 0.90
after the client gets 'ok' acknowledgement from server, while it was not
true in 0.20 (i.e., HBase may have the chance to lose the data even it says
'ok, I got it' to clients). Can someone give me some detailed info?

First, HLog can't seem to be flushed onto disk for each Put operation for
performance reason (i.e., there exists time window when the log of a certain
amout of hlog reside in the RegionServer's memory). If this understanding is
correct, how are we going to assure total durablity?

Secondly, would anyone share the change related to this between 0.90 and
0.20?

Sean



On Mon, Oct 11, 2010 at 1:58 PM, Ryan Rawson <ry...@gmail.com> wrote:

> That is correct.  But we are confident with the new durability changes and
> other things 0.90 will be safer and faster than 0.20.6.
> On Oct 11, 2010 4:51 PM, "Sean Bigdatafun" <se...@gmail.com>
> wrote:
>  > Thanks for clarifying this.
> >
> > But on the other hand, wow... that means that even I like the consistency
> > enhancement in 0.90, I can not enjoy it if I have started using HBase
> 0.20
> > on a production?
> >
> >
> >
> >
> >
> > On Thu, Sep 16, 2010 at 10:49 PM, Stack <st...@duboce.net> wrote:
> >
> >> On Thu, Sep 16, 2010 at 10:22 PM, Todd Lipcon <to...@cloudera.com>
> wrote:
> >> > On Thu, Sep 16, 2010 at 8:51 PM, Stack <st...@duboce.net> wrote:
> >> > What about region ID naming changes? I don't think the new region IDs
> >> would
> >> > work in 0.20.
> >> >
> >>
> >> True.
> >>
> >> So no going back from a 0.89+ to a 0.20 because format of region names
> >> in filesystem has changed.
> >>
> >> St.Ack
> >>
>

Re: HLog and durability question --0.90 and 0.20

[Ryan Rawson <ry...@gmail.com>]

The performance isn't horrible, hundreds of rows/sec and if you use the
write buffer it will speed things up. There is also deferred flush for less
durability but vastly increased performance.
On Oct 13, 2010 6:47 PM, "Stack" <st...@duboce.net> wrote:
> On Mon, Oct 11, 2010 at 5:12 PM, Sean Bigdatafun
> <se...@gmail.com> wrote:
>> Can someone give me a detailed look at the HLog mechanism for 0.90
>> durablity?
>> I recall that HBase committers claim that data will be truly durable in
0.90
>> after the client gets 'ok' acknowledgement from server, while it was not
>> true in 0.20 (i.e., HBase may have the chance to lose the data even it
says
>> 'ok, I got it' to clients). Can someone give me some detailed info?
>>
>
> The issue was that HDFS did not support appends. It would only retain
> data that was in a file that had been properly closed. If a
> regionserver crash while we were appending to the WAL/HLog -- i.e. the
> crash would make it so the writer did not properly close the WAL/HLog
> -- we'd lose effectively all edits that had been applied to the
> WAL/HLog up to the time of the crash.
>
>> First, HLog can't seem to be flushed onto disk for each Put operation for
>> performance reason (i.e., there exists time window when the log of a
certain
>> amout of hlog reside in the RegionServer's memory). If this understanding
is
>> correct, how are we going to assure total durablity?
>>
>
> You can have total durability if you want. You will pay in slower
> performance -- as it is for any database.
>
> St.Ack

Re: HLog and durability question --0.90 and 0.20

[Stack <st...@duboce.net>]

On Mon, Oct 11, 2010 at 5:12 PM, Sean Bigdatafun
<se...@gmail.com> wrote:
> Can someone give me a detailed look at the HLog mechanism for 0.90
> durablity?
> I recall that HBase committers claim that data will be truly durable in 0.90
> after the client gets 'ok' acknowledgement from server, while it was not
> true in 0.20 (i.e., HBase may have the chance to lose the data even it says
> 'ok, I got it' to clients). Can someone give me some detailed info?
>

The issue was that HDFS did not support appends.  It would only retain
data that was in a file that had been properly closed.  If a
regionserver crash while we were appending to the WAL/HLog -- i.e. the
crash would make it so the writer did not properly close the WAL/HLog
-- we'd lose effectively all edits that had been applied to the
WAL/HLog up to the time of the crash.

> First, HLog can't seem to be flushed onto disk for each Put operation for
> performance reason (i.e., there exists time window when the log of a certain
> amout of hlog reside in the RegionServer's memory). If this understanding is
> correct, how are we going to assure total durablity?
>

You can have total durability if you want.  You will pay in slower
performance -- as it is for any database.

St.Ack