You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Tatsuya Kawano <ta...@gmail.com> on 2010/05/17 06:01:46 UTC

HLog durabilty on the current and future Hadoop releases

Hi,

A few days ago, I had a discussion with other Japanese developers on  
hadoop-jp Google group. It was about HLog durability on the recent  
Hadoop releases (0.20.1, 0.20.2)  I never looked at this issue closely  
until then as I was certain to use Hadoop 0.21 from the beginning.

Someone showed us Todd's presentation at HUG March 2010, and we were  
all agreed that in order to solve this issue, we will need to use  
Hadoop trunk or Cloudera CDH3 including HDFS-200 and related patches.

Then I came up with a couple of questions:

1. On Hadoop 0.20.x (without HDFS-200 patch), I must close HLog to  
make it's entries durable, right? While rolling HLog does this, how  
about region server failure?

Someone in the discussion tried this senario. He killed (-9) a region  
server process after a few puts. The HLog was read by HMaster before  
it was closed. HMaster couldn't see any entry in the log and simply  
deleted it. So his lost some puts.

Is this the expected behavior? He used Hadoop 0.20.1 and HBase 0.20.3.

2. On Hadoop trunk, I'd prefer not to hflush() every single put, but  
rely on un-flushed replicas on HDFS nodes, so I can avoid the  
performace penalty. Will this still durable? Will HMaster see un- 
flushed appends right after a region server failure?

Thanks in advance,

-- 
河野 達也
Tatsuya Kawano (mr.)
Tokyo, Japan




Re: HLog durabilty on the current and future Hadoop releases

Posted by Tatsuya Kawano <ta...@gmail.com>.
Thanks Flavio for the bookkeeper info. Good stuff. 

Well, I'm not sure if I want to introduce another complexity to HBase. So upcoming CDH3 will be my choice. But I didn't know about bookkeeper and I'll keep it for something else. 

 - Tatsuya



On 05/17/2010, at 4:51 PM, Flavio Junqueira wrote:

> Hi, Given the topic of this message, I'd like to point out that bookkeeper (HBASE-2315) provides a strong durability guarantee. We sync writes to disk on a quorum of machines. I don't think this feature is currently on the roadmap of hbase, though.
> 
> Thanks,
> -Flavio
> 
> On May 17, 2010, at 6:01 AM, Tatsuya Kawano wrote:
> 
>> 
>> Hi,
>> 
>> A few days ago, I had a discussion with other Japanese developers on
>> hadoop-jp Google group. It was about HLog durability on the recent
>> Hadoop releases (0.20.1, 0.20.2)  I never looked at this issue closely
>> until then as I was certain to use Hadoop 0.21 from the beginning.
>> 
>> Someone showed us Todd's presentation at HUG March 2010, and we were
>> all agreed that in order to solve this issue, we will need to use
>> Hadoop trunk or Cloudera CDH3 including HDFS-200 and related patches.
>> 
>> Then I came up with a couple of questions:
>> 
>> 1. On Hadoop 0.20.x (without HDFS-200 patch), I must close HLog to
>> make it's entries durable, right? While rolling HLog does this, how
>> about region server failure?
>> 
>> Someone in the discussion tried this senario. He killed (-9) a region
>> server process after a few puts. The HLog was read by HMaster before
>> it was closed. HMaster couldn't see any entry in the log and simply
>> deleted it. So his lost some puts.
>> 
>> Is this the expected behavior? He used Hadoop 0.20.1 and HBase 0.20.3.
>> 
>> 2. On Hadoop trunk, I'd prefer not to hflush() every single put, but
>> rely on un-flushed replicas on HDFS nodes, so I can avoid the
>> performace penalty. Will this still durable? Will HMaster see un-
>> flushed appends right after a region server failure?
>> 
>> Thanks in advance,
>> 
>> -- 
>> 河野 達也
>> Tatsuya Kawano (mr.)
>> Tokyo, Japan
>> 
>> 
>> 
> 


Re: HLog durabilty on the current and future Hadoop releases

Posted by Flavio Junqueira <fp...@yahoo-inc.com>.
Hi, Given the topic of this message, I'd like to point out that  
bookkeeper (HBASE-2315) provides a strong durability guarantee. We  
sync writes to disk on a quorum of machines. I don't think this  
feature is currently on the roadmap of hbase, though.

Thanks,
-Flavio

On May 17, 2010, at 6:01 AM, Tatsuya Kawano wrote:

>
> Hi,
>
> A few days ago, I had a discussion with other Japanese developers on
> hadoop-jp Google group. It was about HLog durability on the recent
> Hadoop releases (0.20.1, 0.20.2)  I never looked at this issue closely
> until then as I was certain to use Hadoop 0.21 from the beginning.
>
> Someone showed us Todd's presentation at HUG March 2010, and we were
> all agreed that in order to solve this issue, we will need to use
> Hadoop trunk or Cloudera CDH3 including HDFS-200 and related patches.
>
> Then I came up with a couple of questions:
>
> 1. On Hadoop 0.20.x (without HDFS-200 patch), I must close HLog to
> make it's entries durable, right? While rolling HLog does this, how
> about region server failure?
>
> Someone in the discussion tried this senario. He killed (-9) a region
> server process after a few puts. The HLog was read by HMaster before
> it was closed. HMaster couldn't see any entry in the log and simply
> deleted it. So his lost some puts.
>
> Is this the expected behavior? He used Hadoop 0.20.1 and HBase 0.20.3.
>
> 2. On Hadoop trunk, I'd prefer not to hflush() every single put, but
> rely on un-flushed replicas on HDFS nodes, so I can avoid the
> performace penalty. Will this still durable? Will HMaster see un-
> flushed appends right after a region server failure?
>
> Thanks in advance,
>
> -- 
> 河野 達也
> Tatsuya Kawano (mr.)
> Tokyo, Japan
>
>
>


Re: HLog durabilty on the current and future Hadoop releases

Posted by Ryan Rawson <ry...@gmail.com>.
2010/5/16 Tatsuya Kawano <ta...@gmail.com>:
>
> Hi,
>
> A few days ago, I had a discussion with other Japanese developers on
> hadoop-jp Google group. It was about HLog durability on the recent Hadoop
> releases (0.20.1, 0.20.2)  I never looked at this issue closely until then
> as I was certain to use Hadoop 0.21 from the beginning.
>
> Someone showed us Todd's presentation at HUG March 2010, and we were all
> agreed that in order to solve this issue, we will need to use Hadoop trunk
> or Cloudera CDH3 including HDFS-200 and related patches.
>
> Then I came up with a couple of questions:
>
> 1. On Hadoop 0.20.x (without HDFS-200 patch), I must close HLog to make it's
> entries durable, right? While rolling HLog does this, how about region
> server failure?

The problem is that during failure how do you execute user code?  If
the JVM segfaults hard, we have no opportunity to execute Java code.

>
> Someone in the discussion tried this senario. He killed (-9) a region server
> process after a few puts. The HLog was read by HMaster before it was closed.
> HMaster couldn't see any entry in the log and simply deleted it. So his lost
> some puts.

Yes, this is expected.  Perhaps a refresher is in order:
http://en.wikipedia.org/wiki/SIGKILL

"When sent to a program, SIGKILL causes it to terminate immediately.
In contrast to SIGTERM and SIGINT, this signal cannot be caught or
ignored, and the receiving process cannot perform any clean-up upon
receiving this signal."

As you noted correctly above, you can only replay from a Hadoop file
_if_ the file was correctly closed.

>
> Is this the expected behavior? He used Hadoop 0.20.1 and HBase 0.20.3.
>
> 2. On Hadoop trunk, I'd prefer not to hflush() every single put, but rely on
> un-flushed replicas on HDFS nodes, so I can avoid the performace penalty.
> Will this still durable? Will HMaster see un-flushed appends right after a
> region server failure?

Could you explain how you think hflush() works and what "un-flushed
replicas" means - because none of those concepts you are alluding to
exist in HDFS-200 nor HDFS-265.

>
> Thanks in advance,
>
> --
> 河野 達也
> Tatsuya Kawano (mr.)
> Tokyo, Japan
>
>
>
>

Re: HLog durabilty on the current and future Hadoop releases

Posted by Tatsuya Kawano <ta...@gmail.com>.
Thanks Ryan! Now I got the whole picture. I'm going to go back to the  
Japanese Hadoop community and tell them what you and Todd have told me  
here.

- Tatsuya


On May 18, 2010, at 7:19 AM, Ryan Rawson <ry...@gmail.com> wrote:

> 2010/5/17 Tatsuya Kawano <ta...@gmail.com>:
>>
>> Hi,
>>
>> On 05/17/2010, at 11:50 PM, Todd Lipcon wrote:
>>
>>> 2010/5/16 Tatsuya Kawano <ta...@gmail.com>
>>>
>>>> 2. On Hadoop trunk, I'd prefer not to hflush() every single put,  
>>>> but rely
>>>> on un-flushed replicas on HDFS nodes, so I can avoid the  
>>>> performace penalty.
>>>> Will this still durable? Will HMaster see un-flushed appends  
>>>> right after a
>>>> region server failure?
>>>>
>>>>
>>> If you don't call hflush(), you can still lose edits up to the  
>>> last block
>>> boundary, since hflush is required to persist block locations to the
>>> namenode.
>>>
>>> hflush() does *not* sync to disk - it just makes sure that the  
>>> edits are in
>>> memory on all of the replicas.
>>>
>>> I have some patches staged for CDH3 that will also make the  
>>> performance of
>>> this quite competitive by pipelining hflushes - basically it has  
>>> little to
>>> no effect on throughput, but only a few ms penalty on each write.
>>
>>
>> Thanks Todd. I thought hflush() does sync to disk and I was wrong.  
>> It seems the stuff you put on CDH3 is just the one I wanted!
>>
>> Is your stuff already on the current CDH3 beta?
>>
>>
>>
>> On 05/17/2010, at 2:22 PM, Ryan Rawson wrote:
>>
>>> 2010/5/16 Tatsuya Kawano <ta...@gmail.com>:
>>>> 1. On Hadoop 0.20.x (without HDFS-200 patch), I must close HLog  
>>>> to make it's
>>>> entries durable, right? While rolling HLog does this, how about  
>>>> region
>>>> server failure?
>>>
>>> The problem is that during failure how do you execute user code?  If
>>> the JVM segfaults hard, we have no opportunity to execute Java code.
>>
>> Thanks Ryan. That's right. And OS can crash by a hardware failure  
>> (memory, cpu) and network can be disconnected at anytime. In those  
>> cases, we don't have any opportunity to execute Java code.
>>
>> Is there anything the data node can do after detecting client  
>> timeout?
>
> Things arent quite that simple, the core bug that HDFS-200/265
> attempts to fix is not discarding "blocks under construction".  Right
> now in a HDFS stock, if a file isnt closed, the half-completed blocks
> are just tossed.  Retaining them and managing them is some what
> HDFS-200/265 is about.
>
> Besides just managing under-construction blocks, there is a client
> side flush which pushes data to the server and waits for ACKs before
> returning.  Also there is other client code which when reading from a
> half written file figuring out what to do - normally in a completed
> file all 3 replicas are the same length, but in a crash scenario this
> isnt assured, and you don't just want to give up, you want to recover
> as much data as possible.
>
>> And how much edits I could lose? If a log-roll never happens, is it  
>> going to be up to dfs.block.size (64MB by default)?
>
> Right now?   You lose the most recent HLog.  By default they are
> rotated at 64MB, but you can set it lower to achieve a lower failure
> window.  Also be warned that a smaller log means more HLog files which
> translates into more memstore flushes and generally puts more strain
> on the DFS, thus for really high-update (in terms of bytes/per time)
> set-ups this can cause issues.
>
> We have our prod set up like so:
>    <name>hbase.regionserver.hlog.blocksize</name>
>    <value>2097152</value>
>
> -ryan

Re: HLog durabilty on the current and future Hadoop releases

Posted by Ryan Rawson <ry...@gmail.com>.
2010/5/17 Tatsuya Kawano <ta...@gmail.com>:
>
> Hi,
>
> On 05/17/2010, at 11:50 PM, Todd Lipcon wrote:
>
>> 2010/5/16 Tatsuya Kawano <ta...@gmail.com>
>>
>>> 2. On Hadoop trunk, I'd prefer not to hflush() every single put, but rely
>>> on un-flushed replicas on HDFS nodes, so I can avoid the performace penalty.
>>> Will this still durable? Will HMaster see un-flushed appends right after a
>>> region server failure?
>>>
>>>
>> If you don't call hflush(), you can still lose edits up to the last block
>> boundary, since hflush is required to persist block locations to the
>> namenode.
>>
>> hflush() does *not* sync to disk - it just makes sure that the edits are in
>> memory on all of the replicas.
>>
>> I have some patches staged for CDH3 that will also make the performance of
>> this quite competitive by pipelining hflushes - basically it has little to
>> no effect on throughput, but only a few ms penalty on each write.
>
>
> Thanks Todd. I thought hflush() does sync to disk and I was wrong. It seems the stuff you put on CDH3 is just the one I wanted!
>
> Is your stuff already on the current CDH3 beta?
>
>
>
> On 05/17/2010, at 2:22 PM, Ryan Rawson wrote:
>
>> 2010/5/16 Tatsuya Kawano <ta...@gmail.com>:
>>> 1. On Hadoop 0.20.x (without HDFS-200 patch), I must close HLog to make it's
>>> entries durable, right? While rolling HLog does this, how about region
>>> server failure?
>>
>> The problem is that during failure how do you execute user code?  If
>> the JVM segfaults hard, we have no opportunity to execute Java code.
>
> Thanks Ryan. That's right. And OS can crash by a hardware failure (memory, cpu) and network can be disconnected at anytime. In those cases, we don't have any opportunity to execute Java code.
>
> Is there anything the data node can do after detecting client timeout?

Things arent quite that simple, the core bug that HDFS-200/265
attempts to fix is not discarding "blocks under construction".  Right
now in a HDFS stock, if a file isnt closed, the half-completed blocks
are just tossed.  Retaining them and managing them is some what
HDFS-200/265 is about.

Besides just managing under-construction blocks, there is a client
side flush which pushes data to the server and waits for ACKs before
returning.  Also there is other client code which when reading from a
half written file figuring out what to do - normally in a completed
file all 3 replicas are the same length, but in a crash scenario this
isnt assured, and you don't just want to give up, you want to recover
as much data as possible.

> And how much edits I could lose? If a log-roll never happens, is it going to be up to dfs.block.size (64MB by default)?

Right now?   You lose the most recent HLog.  By default they are
rotated at 64MB, but you can set it lower to achieve a lower failure
window.  Also be warned that a smaller log means more HLog files which
translates into more memstore flushes and generally puts more strain
on the DFS, thus for really high-update (in terms of bytes/per time)
set-ups this can cause issues.

We have our prod set up like so:
    <name>hbase.regionserver.hlog.blocksize</name>
    <value>2097152</value>

-ryan

Re: HLog durabilty on the current and future Hadoop releases

Posted by Todd Lipcon <to...@cloudera.com>.
2010/5/17 Tatsuya Kawano <ta...@gmail.com>

>
> Hi,
>
> On 05/17/2010, at 11:50 PM, Todd Lipcon wrote:
>
> > 2010/5/16 Tatsuya Kawano <ta...@gmail.com>
> >
> >> 2. On Hadoop trunk, I'd prefer not to hflush() every single put, but
> rely
> >> on un-flushed replicas on HDFS nodes, so I can avoid the performace
> penalty.
> >> Will this still durable? Will HMaster see un-flushed appends right after
> a
> >> region server failure?
> >>
> >>
> > If you don't call hflush(), you can still lose edits up to the last block
> > boundary, since hflush is required to persist block locations to the
> > namenode.
> >
> > hflush() does *not* sync to disk - it just makes sure that the edits are
> in
> > memory on all of the replicas.
> >
> > I have some patches staged for CDH3 that will also make the performance
> of
> > this quite competitive by pipelining hflushes - basically it has little
> to
> > no effect on throughput, but only a few ms penalty on each write.
>
>
> Thanks Todd. I thought hflush() does sync to disk and I was wrong. It seems
> the stuff you put on CDH3 is just the one I wanted!
>
> Is your stuff already on the current CDH3 beta?
>

Not yet - still undergoing testing in my "lab" cluster. We should have a
beta out next month.


>
>
>
> On 05/17/2010, at 2:22 PM, Ryan Rawson wrote:
>
> > 2010/5/16 Tatsuya Kawano <ta...@gmail.com>:
> >> 1. On Hadoop 0.20.x (without HDFS-200 patch), I must close HLog to make
> it's
> >> entries durable, right? While rolling HLog does this, how about region
> >> server failure?
> >
> > The problem is that during failure how do you execute user code?  If
> > the JVM segfaults hard, we have no opportunity to execute Java code.
>
> Thanks Ryan. That's right. And OS can crash by a hardware failure (memory,
> cpu) and network can be disconnected at anytime. In those cases, we don't
> have any opportunity to execute Java code.
>
> Is there anything the data node can do after detecting client timeout?
>
> And how much edits I could lose? If a log-roll never happens, is it going
> to be up to dfs.block.size (64MB by default)?
>
>
> Thanks,
> Tatsuya
>
> --
> 河野 達也
> Tatsuya Kawano (mr.)
> Tokyo, Japan
>
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Re: HLog durabilty on the current and future Hadoop releases

Posted by Tatsuya Kawano <ta...@gmail.com>.
Hi, 

On 05/17/2010, at 11:50 PM, Todd Lipcon wrote:

> 2010/5/16 Tatsuya Kawano <ta...@gmail.com>
> 
>> 2. On Hadoop trunk, I'd prefer not to hflush() every single put, but rely
>> on un-flushed replicas on HDFS nodes, so I can avoid the performace penalty.
>> Will this still durable? Will HMaster see un-flushed appends right after a
>> region server failure?
>> 
>> 
> If you don't call hflush(), you can still lose edits up to the last block
> boundary, since hflush is required to persist block locations to the
> namenode.
> 
> hflush() does *not* sync to disk - it just makes sure that the edits are in
> memory on all of the replicas.
> 
> I have some patches staged for CDH3 that will also make the performance of
> this quite competitive by pipelining hflushes - basically it has little to
> no effect on throughput, but only a few ms penalty on each write.


Thanks Todd. I thought hflush() does sync to disk and I was wrong. It seems the stuff you put on CDH3 is just the one I wanted! 

Is your stuff already on the current CDH3 beta?



On 05/17/2010, at 2:22 PM, Ryan Rawson wrote:

> 2010/5/16 Tatsuya Kawano <ta...@gmail.com>:
>> 1. On Hadoop 0.20.x (without HDFS-200 patch), I must close HLog to make it's
>> entries durable, right? While rolling HLog does this, how about region
>> server failure?
> 
> The problem is that during failure how do you execute user code?  If
> the JVM segfaults hard, we have no opportunity to execute Java code.

Thanks Ryan. That's right. And OS can crash by a hardware failure (memory, cpu) and network can be disconnected at anytime. In those cases, we don't have any opportunity to execute Java code. 

Is there anything the data node can do after detecting client timeout? 

And how much edits I could lose? If a log-roll never happens, is it going to be up to dfs.block.size (64MB by default)? 


Thanks, 
Tatsuya

-- 
河野 達也
Tatsuya Kawano (mr.)
Tokyo, Japan



Re: HLog durabilty on the current and future Hadoop releases

Posted by Todd Lipcon <to...@cloudera.com>.
2010/5/16 Tatsuya Kawano <ta...@gmail.com>

> 2. On Hadoop trunk, I'd prefer not to hflush() every single put, but rely
> on un-flushed replicas on HDFS nodes, so I can avoid the performace penalty.
> Will this still durable? Will HMaster see un-flushed appends right after a
> region server failure?
>
>
If you don't call hflush(), you can still lose edits up to the last block
boundary, since hflush is required to persist block locations to the
namenode.

hflush() does *not* sync to disk - it just makes sure that the edits are in
memory on all of the replicas.

I have some patches staged for CDH3 that will also make the performance of
this quite competitive by pipelining hflushes - basically it has little to
no effect on throughput, but only a few ms penalty on each write.

-Todd


-- 
Todd Lipcon
Software Engineer, Cloudera