You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Something Something <lu...@yahoo.com> on 2009/09/30 18:22:09 UTC
HBase configuration...
Hello,
I noticed that if I start HBase in "standalone" mode it creates a table in memory. In other words, after rebooting the machine, the table goes away. I would like to persist the table to a local file system. How can I do that? Do I have to use "Psuedo Distribution" mode for this?
Please help. Thanks.
Re: HBase configuration...
Posted by stack <st...@duboce.net>.
I'd suggest you not do 'proof of concept' on standalone hbase loading up
100M rows. The picture you'll get will be skewed. Do 1M small rows in
standalone. Get yourself a cluster north of 5 nodes if you are serious
about your hbase evaluation. Read over the wiki and mailing lists to learn
from the experience of others (Read the 'Getting Started' too).
Let us know if you need help.
Yours,
St.Ack
On Wed, Sep 30, 2009 at 3:33 PM, Something Something <luckyguy2050@yahoo.com
> wrote:
> Hmmm.. Interesting... I could swear the 2nd time around I had run
> stop-hbase.sh before rebooting. But may be I didn't wait long enough. My
> job adds over 100 million rows to a table. I have plenty of free space on
> my hard drive, so I am assuming this will all work in "standalone" mode,
> correct? (Just doing a proof of concept for now.)
>
> In any case, thanks for the reply. It sounds like the tables should get
> persisted to disk. Will try once again. Thanks.
>
> --- On Wed, 9/30/09, stack <st...@duboce.net> wrote:
>
> From: stack <st...@duboce.net>
> Subject: Re: HBase configuration...
> To: general@hadoop.apache.org
> Date: Wednesday, September 30, 2009, 2:42 PM
>
> Shut it down cleanly (./bin/stop-hbase.sh). My guess is that you are
> killing it before it has chance to flush its in-memory state.
>
> HBase questions will get more timely response if posted to the hbase lists
> (see hbase.org).
>
> Yours,
> St.Ack St.Ack
>
>
> On Wed, Sep 30, 2009 at 9:22 AM, Something Something <
> luckyguy2050@yahoo.com
> > wrote:
>
> > Hello,
> >
> > I noticed that if I start HBase in "standalone" mode it creates a table
> in
> > memory. In other words, after rebooting the machine, the table goes
> away.
> > I would like to persist the table to a local file system. How can I do
> > that? Do I have to use "Psuedo Distribution" mode for this?
> >
> > Please help. Thanks.
> >
> >
> >
> >
> >
>
>
>
>
>
Re: HBase configuration...
Posted by Kevin Peterson <kp...@biz360.com>.
On Wed, Sep 30, 2009 at 3:33 PM, Something Something <luckyguy2050@yahoo.com
> wrote:
> Hmmm.. Interesting... I could swear the 2nd time around I had run
> stop-hbase.sh before rebooting. But may be I didn't wait long enough. My
> job adds over 100 million rows to a table. I have plenty of free space on
> my hard drive, so I am assuming this will all work in "standalone" mode,
> correct? (Just doing a proof of concept for now.)
>
> By default it gets saved into /tmp, which on many machines gets cleared out
every night or after every boot.
Set hbase.rootdir to something else (writable) to ensure it isn't cleared on
reboot.
Re: HBase configuration...
Posted by Ryan Rawson <ry...@gmail.com>.
That seems reasonable, although the nice thing about /tmp is how
universal it is :-)
On Thu, Oct 1, 2009 at 1:05 AM, stephen mulcahy
<st...@deri.org> wrote:
> Ryan Rawson wrote:
>>
>> yes, the standalone version uses file:///tmp to store data, to allow
>> people to get up and running without HDFS. And you are exactly
>> correct, with /tmp being cleaned on reboot on many distros. Grep for
>> /tmp in hbase-default.xml to find the setting.
>
> Would it be better to default this directory to /var/tmp which would achieve
> the same goal of allowing people to quickly get up and running but might
> avoid some of the confusion of their HDFS disappearing on reboot (speaking
> as one recently confused in such a manner ;)
>
> -stephen
>
> --
> Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
> NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
> http://di2.deri.ie http://webstar.deri.ie http://sindice.com
>
Re: HBase configuration...
Posted by stephen mulcahy <st...@deri.org>.
Ryan Rawson wrote:
> yes, the standalone version uses file:///tmp to store data, to allow
> people to get up and running without HDFS. And you are exactly
> correct, with /tmp being cleaned on reboot on many distros. Grep for
> /tmp in hbase-default.xml to find the setting.
Would it be better to default this directory to /var/tmp which would
achieve the same goal of allowing people to quickly get up and running
but might avoid some of the confusion of their HDFS disappearing on
reboot (speaking as one recently confused in such a manner ;)
-stephen
--
Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
http://di2.deri.ie http://webstar.deri.ie http://sindice.com
Re: HBase configuration...
Posted by Ryan Rawson <ry...@gmail.com>.
yes, the standalone version uses file:///tmp to store data, to allow
people to get up and running without HDFS. And you are exactly
correct, with /tmp being cleaned on reboot on many distros. Grep for
/tmp in hbase-default.xml to find the setting.
-ryan
On Thu, Oct 1, 2009 at 12:47 AM, stephen mulcahy
<st...@deri.org> wrote:
> Hi,
>
> I could be way off the mark here - but if you are running HBase on top of
> Hadoop in the default configuration then it may be using /tmp for HDFS. This
> gets wiped on reboot which would cause the problems you are seeing.
>
> -stephen
>
> stack wrote:
>>
>> Shut it down cleanly (./bin/stop-hbase.sh). My guess is that you are
>> killing it before it has chance to flush its in-memory state.
>>
>> HBase questions will get more timely response if posted to the hbase lists
>> (see hbase.org).
>>
>> Yours,
>> St.Ack St.Ack
>>
>>
>> On Wed, Sep 30, 2009 at 9:22 AM, Something Something
>> <luckyguy2050@yahoo.com
>>>
>>> wrote:
>>
>>> Hello,
>>>
>>> I noticed that if I start HBase in "standalone" mode it creates a table
>>> in
>>> memory. In other words, after rebooting the machine, the table goes
>>> away.
>>> I would like to persist the table to a local file system. How can I do
>>> that? Do I have to use "Psuedo Distribution" mode for this?
>>>
>>> Please help. Thanks.
>>>
>>>
>>>
>>>
>>>
>>
>
>
> --
> Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
> NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
> http://di2.deri.ie http://webstar.deri.ie http://sindice.com
>
Re: HBase configuration...
Posted by stephen mulcahy <st...@deri.org>.
Hi,
I could be way off the mark here - but if you are running HBase on top
of Hadoop in the default configuration then it may be using /tmp for
HDFS. This gets wiped on reboot which would cause the problems you are
seeing.
-stephen
stack wrote:
> Shut it down cleanly (./bin/stop-hbase.sh). My guess is that you are
> killing it before it has chance to flush its in-memory state.
>
> HBase questions will get more timely response if posted to the hbase lists
> (see hbase.org).
>
> Yours,
> St.Ack St.Ack
>
>
> On Wed, Sep 30, 2009 at 9:22 AM, Something Something <luckyguy2050@yahoo.com
>> wrote:
>
>> Hello,
>>
>> I noticed that if I start HBase in "standalone" mode it creates a table in
>> memory. In other words, after rebooting the machine, the table goes away.
>> I would like to persist the table to a local file system. How can I do
>> that? Do I have to use "Psuedo Distribution" mode for this?
>>
>> Please help. Thanks.
>>
>>
>>
>>
>>
>
--
Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
http://di2.deri.ie http://webstar.deri.ie http://sindice.com
Re: HBase configuration...
Posted by Something Something <lu...@yahoo.com>.
Hmmm.. Interesting... I could swear the 2nd time around I had run stop-hbase.sh before rebooting. But may be I didn't wait long enough. My job adds over 100 million rows to a table. I have plenty of free space on my hard drive, so I am assuming this will all work in "standalone" mode, correct? (Just doing a proof of concept for now.)
In any case, thanks for the reply. It sounds like the tables should get persisted to disk. Will try once again. Thanks.
--- On Wed, 9/30/09, stack <st...@duboce.net> wrote:
From: stack <st...@duboce.net>
Subject: Re: HBase configuration...
To: general@hadoop.apache.org
Date: Wednesday, September 30, 2009, 2:42 PM
Shut it down cleanly (./bin/stop-hbase.sh). My guess is that you are
killing it before it has chance to flush its in-memory state.
HBase questions will get more timely response if posted to the hbase lists
(see hbase.org).
Yours,
St.Ack St.Ack
On Wed, Sep 30, 2009 at 9:22 AM, Something Something <luckyguy2050@yahoo.com
> wrote:
> Hello,
>
> I noticed that if I start HBase in "standalone" mode it creates a table in
> memory. In other words, after rebooting the machine, the table goes away.
> I would like to persist the table to a local file system. How can I do
> that? Do I have to use "Psuedo Distribution" mode for this?
>
> Please help. Thanks.
>
>
>
>
>
Re: HBase configuration...
Posted by Something Something <lu...@yahoo.com>.
Hmmm.. Interesting... I could swear the 2nd time around I had run stop-hbase.sh before rebooting. But may be I didn't wait long enough. My job adds over 100 million rows to a table. I have plenty of free space on my hard drive, so I am assuming this will all work in "standalone" mode, correct? (Just doing a proof of concept for now.)
In any case, thanks for the reply. It sounds like the tables should get persisted to disk. Will try once again. Thanks.
--- On Wed, 9/30/09, stack <st...@duboce.net> wrote:
From: stack <st...@duboce.net>
Subject: Re: HBase configuration...
To: general@hadoop.apache.org
Date: Wednesday, September 30, 2009, 2:42 PM
Shut it down cleanly (./bin/stop-hbase.sh). My guess is that you are
killing it before it has chance to flush its in-memory state.
HBase questions will get more timely response if posted to the hbase lists
(see hbase.org).
Yours,
St.Ack St.Ack
On Wed, Sep 30, 2009 at 9:22 AM, Something Something <luckyguy2050@yahoo.com
> wrote:
> Hello,
>
> I noticed that if I start HBase in "standalone" mode it creates a table in
> memory. In other words, after rebooting the machine, the table goes away.
> I would like to persist the table to a local file system. How can I do
> that? Do I have to use "Psuedo Distribution" mode for this?
>
> Please help. Thanks.
>
>
>
>
>
Re: HBase configuration...
Posted by stack <st...@duboce.net>.
Shut it down cleanly (./bin/stop-hbase.sh). My guess is that you are
killing it before it has chance to flush its in-memory state.
HBase questions will get more timely response if posted to the hbase lists
(see hbase.org).
Yours,
St.Ack St.Ack
On Wed, Sep 30, 2009 at 9:22 AM, Something Something <luckyguy2050@yahoo.com
> wrote:
> Hello,
>
> I noticed that if I start HBase in "standalone" mode it creates a table in
> memory. In other words, after rebooting the machine, the table goes away.
> I would like to persist the table to a local file system. How can I do
> that? Do I have to use "Psuedo Distribution" mode for this?
>
> Please help. Thanks.
>
>
>
>
>