Is there a way to replicate root and meta table in HBase?

[Gen Liu <ge...@zynga.com>]

Hi,

We are running into a case that if the region server that serves meta table is down, all request will timeouts because region lookup is not available. At this time, master is also not able to update meta table.
It seems that regions that serve root and meta are the single point of failure in HBase.
Is there a way to get rid of it? Does HBase give a higher recover priority to meta and root table?

Thanks.
Gen

Re: Is there a way to replicate root and meta table in HBase?

[Stas Maksimov <ma...@gmail.com>]

Just today I saw this mentioned in the docs. They said they deliberately
don't replicate those, otherwise "it gets very messy".

Stas

On Tue, Sep 4, 2012 at 10:52 PM, Gen Liu <ge...@zynga.com> wrote:

> Hi,
>
> We are running into a case that if the region server that serves meta
> table is down, all request will timeouts because region lookup is not
> available. At this time, master is also not able to update meta table.
> It seems that regions that serve root and meta are the single point of
> failure in HBase.
> Is there a way to get rid of it? Does HBase give a higher recover priority
> to meta and root table?
>
> Thanks.
> Gen
>

Re: Is there a way to replicate root and meta table in HBase?

[Gen Liu <ge...@zynga.com>]

On 9/4/12 3:07 PM, "Stack" <st...@duboce.net> wrote:

>On Tue, Sep 4, 2012 at 2:52 PM, Gen Liu <ge...@zynga.com> wrote:
>> We are running into a case that if the region server that serves meta
>>table is down, all request will timeouts because region lookup is not
>>available.
>
>Only requests to .META. fail (and most of the time, .META. info is
>cached so should be relatively rare to do .META. lookups).  It should
>not be all requests.

We get a lot of region lookup error in the client side:
(zlive-hbase-08.int.zynga.com is not dead, we killed another server)
2012-09-04 14:28:11,829 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
: locateRegionInMeta parentTable=-ROOT-,
metaLocation={region=-ROOT-,,0.70236052,
hostname=zlive-hbase-08.int.zynga.com, port=60020}, attempt=8 of 10
failed; retrying after sleep of 16000 because: Connection refused

>
>> It seems that regions that serve root and meta are the single point of
>>failure in HBase.
>
>They can be offline if a server crashes but they should be back on
>line soon enough; is this not your experience?

We set hbase.regionserver.maxlogs=256 to enable big memstore flush to
lower compaction stress, so the log split takes about 5-10 minutes. I
think META will come back after the log split.
Is there a way to specify where should HBase put root and meta table?

>
>> Is there a way to get rid of it? Does HBase give a higher recover
>>priority to meta and root table?
>>
>
>HBase gets .META. and -ROOT- back on line ahead of all other regions, yes.
>
>St.Ack

Re: Is there a way to replicate root and meta table in HBase?

[Stack <st...@duboce.net>]

On Tue, Sep 4, 2012 at 2:52 PM, Gen Liu <ge...@zynga.com> wrote:
> We are running into a case that if the region server that serves meta table is down, all request will timeouts because region lookup is not available.

Only requests to .META. fail (and most of the time, .META. info is
cached so should be relatively rare to do .META. lookups).  It should
not be all requests.

> It seems that regions that serve root and meta are the single point of failure in HBase.

They can be offline if a server crashes but they should be back on
line soon enough; is this not your experience?

> Is there a way to get rid of it? Does HBase give a higher recover priority to meta and root table?
>

HBase gets .META. and -ROOT- back on line ahead of all other regions, yes.

St.Ack