You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Yifeng Jiang <up...@gmail.com> on 2011/12/24 03:59:35 UTC
What's the scenario of manual hlog splitting?
Hi,
As mentioned in HBase Book, we can force a manual hlog splitting by:
hbase org.apache.hadoop.hbase.regionserver.wal.HLog --split hdfs://example.org:8020/hbase/.logs/example.org,60020,1283516293161/
What's the use case of this manual splitting?
If RS is crashed, the hlog splitting will be triggered by master automatically.
While, force a hlog splitting shuts down the RS if the RS is online.
Thanks,
Yifeng
Re: What's the scenario of manual hlog splitting?
Posted by Yifeng Jiang <up...@gmail.com>.
Thanks Jon,
Just for parallel log splitting.
I got it.
Yifeng
On Dec 25, 2011, at 12:39 AM, Jonathan Hsieh wrote:
> Yifeng,
>
> HLog splitting is used to when shutting down a region server or to recover
> log edits after a region server goes down uncleanly.
>
> Prior to 0.92, if many region servers go down uncleanly (maybe hdfs went
> away under hbase), only a single process would do all log splitting in to
> recover. If you have a lot of nodes, each has a hlog to split this can
> take a long time (100 nodes = 100x the time). If you have many nodes going
> down uncleanly you could manually use the HLog --split mechanism you
> describe to split all the logs in parallel on many machines before
> restarting hbase.
>
> As of 0.92, this hlog splitting is automatically distributed -- the feature
> is called distributed log splitting. This farms work out the the nodes and
> should significantly speedup up recovery in these scenarios.
>
> Jon.
>
> On Fri, Dec 23, 2011 at 6:59 PM, Yifeng Jiang <up...@gmail.com> wrote:
>
>> Hi,
>>
>> As mentioned in HBase Book, we can force a manual hlog splitting by:
>> hbase org.apache.hadoop.hbase.regionserver.wal.HLog --split hdfs://
>> example.org:8020/hbase/.logs/example.org,60020,1283516293161/
>>
>> What's the use case of this manual splitting?
>> If RS is crashed, the hlog splitting will be triggered by master
>> automatically.
>> While, force a hlog splitting shuts down the RS if the RS is online.
>>
>> Thanks,
>> Yifeng
>
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
Re: What's the scenario of manual hlog splitting?
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Yifeng,
HLog splitting is used to when shutting down a region server or to recover
log edits after a region server goes down uncleanly.
Prior to 0.92, if many region servers go down uncleanly (maybe hdfs went
away under hbase), only a single process would do all log splitting in to
recover. If you have a lot of nodes, each has a hlog to split this can
take a long time (100 nodes = 100x the time). If you have many nodes going
down uncleanly you could manually use the HLog --split mechanism you
describe to split all the logs in parallel on many machines before
restarting hbase.
As of 0.92, this hlog splitting is automatically distributed -- the feature
is called distributed log splitting. This farms work out the the nodes and
should significantly speedup up recovery in these scenarios.
Jon.
On Fri, Dec 23, 2011 at 6:59 PM, Yifeng Jiang <up...@gmail.com> wrote:
> Hi,
>
> As mentioned in HBase Book, we can force a manual hlog splitting by:
> hbase org.apache.hadoop.hbase.regionserver.wal.HLog --split hdfs://
> example.org:8020/hbase/.logs/example.org,60020,1283516293161/
>
> What's the use case of this manual splitting?
> If RS is crashed, the hlog splitting will be triggered by master
> automatically.
> While, force a hlog splitting shuts down the RS if the RS is online.
>
> Thanks,
> Yifeng
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com