You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Buttler, David" <bu...@llnl.gov> on 2011/03/01 00:59:58 UTC
RE: Question on upgrade chapter
Hi Guys,
Thanks for the clarifications. I did indeed mistype what I did. I scanned -ROOT- (if you are curious, it is appended below)
Since I am still on 0.20.6 until I figure out how to make my code work with 0.90.1 on my test cluster, how should I go about fixing the MEMSTORE_FLUSHSIZE?
Thanks for the help,
Dave
---
hbase(main):004:0> scan '-ROOT-'
ROW COLUMN+CELL
.META.,,1 column=info:regioninfo, timestamp=1298662009676, value=REGION => {NAME => '.META.
,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.M
ETA.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384', FAMILIES => [{NAME => 'h
istorian', VERSIONS => '2147483647', COMPRESSION => 'NONE', TTL => '604800', BLOC
KSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', V
ERSIONS => '10', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '8192',
IN_MEMORY => 'true', BLOCKCACHE => 'true'}]}}
.META.,,1 column=info:server, timestamp=1298681408958, value=10.0.1.2:60020
.META.,,1 column=info:serverstartcode, timestamp=1298681408958, value=1298681408122
-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
Sent: Monday, February 28, 2011 2:57 PM
To: user@hbase.apache.org
Subject: Re: Question on upgrade chapter
Maybe it's a typo in your email, but it says you need to scan -ROOT-
to check your .META.'s MEMSTORE_FLUSHSIZE (you seem to imply that you
scanned .META.)
If MEMSTORE_FLUSHSIZE wasn't changed (default is 64MB) then it won't
be printed. I'm going to change the documentation to make it clearer.
Finally, I don't know how you got a table with
MEMSTORE_FLUSHSIZE=16384 but that's probably a very bad performance
issue.
J-D
On Mon, Feb 28, 2011 at 2:26 PM, Buttler, David <bu...@llnl.gov> wrote:
> Hi,
>
> In
> http://hbase.apache.org/upgrading.html
> it says that I should scan my .META. table to check my MEMSTORE_SIZE to ensure that it is not 16MB.
> It is not at all clear what the correct value should be (e.g., what set_meta_memstore_size.rb will change it to)
>
> Also, I do not even see this entry in my table. The closest thing I see is MEMSTORE_FLUSHSIZE=16384.
> I assume this means that I do not have this particular issue. If that is the case, it might be a nice addendum to say that MEMSTORE_SIZE != MEMSTORE_FLUSHSIZE. And maybe also put the value for 16MB (16777216) in parenthesis to be very clear.
>
> Thanks,
> Dave
>
Re: Zookeeper / Hbase
Posted by Mark Kerzner <ma...@gmail.com>.
Hey, Joseph, I am also interested in Avatar - and can one make it work :)
Please share if you can.
Mark
On Tue, Mar 1, 2011 at 12:23 PM, Joseph Coleman <
joe.coleman@infinitecampus.com> wrote:
> Thanks, I am only doing 3 servers total for running through the setup for
> comfort sakes before my production gear gets here. I am looking at a
> single Master HDFS server will do Avatar if I can figure it out. The have
> a 10 node data cluster for HDFS and Hbase and a 3 node cluster for
> Zookeeper. Because at some point we way me 20 plus data nodes by years
> end.. Or am I better off leaving Hbase to manage ZK? If so what point do I
> separate out ZK to be its own management piece.
>
>
>
>
>
>
> On 3/1/11 9:15 AM, "Lars George" <la...@gmail.com> wrote:
>
> >Hi Joseph,
> >
> >You are talking about a full distributed setup - just all with single
> >nodes? So your ZooKeeper is started and maintained by you as well
> >separately? If so, then sure you can run it on your own. Well, even
> >with HBase you can run this on your own using the supplied version
> >that comes with HBase, see http://hbase.apache.org/book.html#zookeeper
> >for all the gory details.
> >
> >Lars
> >
> >On Tue, Mar 1, 2011 at 3:41 PM, Joseph Coleman
> ><jo...@infinitecampus.com> wrote:
> >> Question when setting up the zookeeper master node can I have just
> >> zookeeper on that box or do I need to have Hbase and hadoop installed on
> >> there as well? I am just doing a 3 node setup for an exercise before my
> >> production gear gets here.. So I have a working HDFS cluster 1 master
> >>and
> >> one slave just trying to setup Hbase and Zookeeper with The zookeeper
> >> master also serving as the Hbase master and the on one HDFS slave also
> >> being my Hbase slave..
> >>
> >>
>
>
Re: Zookeeper / Hbase
Posted by Lars George <la...@gmail.com>.
Hi Joseph,
As Dave says, you could always use HBase to manage ZooKeeper. If you
need it for other things as well, and the AvatarNode is one of those,
then you have to make sure you set
HBASE_MANAGES_ZK=false
in the hbase-env.sh but still use the HBase scripts to start and stop
the stand-alone ZooKeeper ensemble, including running the nodes on
shared servers, such as the master etc.
Lars
On Tue, Mar 1, 2011 at 7:23 PM, Joseph Coleman
<jo...@infinitecampus.com> wrote:
> Thanks, I am only doing 3 servers total for running through the setup for
> comfort sakes before my production gear gets here. I am looking at a
> single Master HDFS server will do Avatar if I can figure it out. The have
> a 10 node data cluster for HDFS and Hbase and a 3 node cluster for
> Zookeeper. Because at some point we way me 20 plus data nodes by years
> end.. Or am I better off leaving Hbase to manage ZK? If so what point do I
> separate out ZK to be its own management piece.
>
>
>
>
>
>
> On 3/1/11 9:15 AM, "Lars George" <la...@gmail.com> wrote:
>
>>Hi Joseph,
>>
>>You are talking about a full distributed setup - just all with single
>>nodes? So your ZooKeeper is started and maintained by you as well
>>separately? If so, then sure you can run it on your own. Well, even
>>with HBase you can run this on your own using the supplied version
>>that comes with HBase, see http://hbase.apache.org/book.html#zookeeper
>>for all the gory details.
>>
>>Lars
>>
>>On Tue, Mar 1, 2011 at 3:41 PM, Joseph Coleman
>><jo...@infinitecampus.com> wrote:
>>> Question when setting up the zookeeper master node can I have just
>>> zookeeper on that box or do I need to have Hbase and hadoop installed on
>>> there as well? I am just doing a 3 node setup for an exercise before my
>>> production gear gets here.. So I have a working HDFS cluster 1 master
>>>and
>>> one slave just trying to setup Hbase and Zookeeper with The zookeeper
>>> master also serving as the Hbase master and the on one HDFS slave also
>>> being my Hbase slave..
>>>
>>>
>
>
RE: Zookeeper / Hbase
Posted by "Buttler, David" <bu...@llnl.gov>.
I really like the way HBase manages zookeeper. It seems much more intuitive to me than the native zookeeper configuration. For my cluster I use zookeeper for a couple of different tasks (like hbase, solr cloud, and other home grown things). I manage Zookeeper using a slightly different version of the hbase scripts. This allows me to start and stop hbase without affecting the zookeeper cluster -- but I do have to remember to stop hbase if I need to stop zookeeper.
In my opinion, HBase improves zookeeper. [although I would be happy to hear why I am missing something from the standard zookeeper config]
Dave
-----Original Message-----
From: Lars George [mailto:lars.george@gmail.com]
Sent: Tuesday, March 01, 2011 7:15 AM
To: user@hbase.apache.org
Subject: Re: Zookeeper / Hbase
Hi Joseph,
You are talking about a full distributed setup - just all with single
nodes? So your ZooKeeper is started and maintained by you as well
separately? If so, then sure you can run it on your own. Well, even
with HBase you can run this on your own using the supplied version
that comes with HBase, see http://hbase.apache.org/book.html#zookeeper
for all the gory details.
Lars
On Tue, Mar 1, 2011 at 3:41 PM, Joseph Coleman
<jo...@infinitecampus.com> wrote:
> Question when setting up the zookeeper master node can I have just
> zookeeper on that box or do I need to have Hbase and hadoop installed on
> there as well? I am just doing a 3 node setup for an exercise before my
> production gear gets here.. So I have a working HDFS cluster 1 master and
> one slave just trying to setup Hbase and Zookeeper with The zookeeper
> master also serving as the Hbase master and the on one HDFS slave also
> being my Hbase slave..
>
>
Re: Zookeeper / Hbase
Posted by Joseph Coleman <jo...@infinitecampus.com>.
Thanks, I am only doing 3 servers total for running through the setup for
comfort sakes before my production gear gets here. I am looking at a
single Master HDFS server will do Avatar if I can figure it out. The have
a 10 node data cluster for HDFS and Hbase and a 3 node cluster for
Zookeeper. Because at some point we way me 20 plus data nodes by years
end.. Or am I better off leaving Hbase to manage ZK? If so what point do I
separate out ZK to be its own management piece.
On 3/1/11 9:15 AM, "Lars George" <la...@gmail.com> wrote:
>Hi Joseph,
>
>You are talking about a full distributed setup - just all with single
>nodes? So your ZooKeeper is started and maintained by you as well
>separately? If so, then sure you can run it on your own. Well, even
>with HBase you can run this on your own using the supplied version
>that comes with HBase, see http://hbase.apache.org/book.html#zookeeper
>for all the gory details.
>
>Lars
>
>On Tue, Mar 1, 2011 at 3:41 PM, Joseph Coleman
><jo...@infinitecampus.com> wrote:
>> Question when setting up the zookeeper master node can I have just
>> zookeeper on that box or do I need to have Hbase and hadoop installed on
>> there as well? I am just doing a 3 node setup for an exercise before my
>> production gear gets here.. So I have a working HDFS cluster 1 master
>>and
>> one slave just trying to setup Hbase and Zookeeper with The zookeeper
>> master also serving as the Hbase master and the on one HDFS slave also
>> being my Hbase slave..
>>
>>
Re: Zookeeper / Hbase
Posted by Lars George <la...@gmail.com>.
Hi Joseph,
You are talking about a full distributed setup - just all with single
nodes? So your ZooKeeper is started and maintained by you as well
separately? If so, then sure you can run it on your own. Well, even
with HBase you can run this on your own using the supplied version
that comes with HBase, see http://hbase.apache.org/book.html#zookeeper
for all the gory details.
Lars
On Tue, Mar 1, 2011 at 3:41 PM, Joseph Coleman
<jo...@infinitecampus.com> wrote:
> Question when setting up the zookeeper master node can I have just
> zookeeper on that box or do I need to have Hbase and hadoop installed on
> there as well? I am just doing a 3 node setup for an exercise before my
> production gear gets here.. So I have a working HDFS cluster 1 master and
> one slave just trying to setup Hbase and Zookeeper with The zookeeper
> master also serving as the Hbase master and the on one HDFS slave also
> being my Hbase slave..
>
>
Re: Question on upgrade chapter
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Actually, the upgrade doc is wrong. It's 16KB not 16MB so that value
is right, and it's the one you want for 0.20.6 since it lowers greatly
the chance of dataloss for .META. which is pretty catastrophic.
Only change it when you do upgrade.
J-D
On Mon, Feb 28, 2011 at 3:59 PM, Buttler, David <bu...@llnl.gov> wrote:
> Hi Guys,
> Thanks for the clarifications. I did indeed mistype what I did. I scanned -ROOT- (if you are curious, it is appended below)
>
> Since I am still on 0.20.6 until I figure out how to make my code work with 0.90.1 on my test cluster, how should I go about fixing the MEMSTORE_FLUSHSIZE?
>
> Thanks for the help,
> Dave
>
>
>
>
>
>
>
>
>
>
> ---
> hbase(main):004:0> scan '-ROOT-'
> ROW COLUMN+CELL
> .META.,,1 column=info:regioninfo, timestamp=1298662009676, value=REGION => {NAME => '.META.
> ,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.M
> ETA.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384', FAMILIES => [{NAME => 'h
> istorian', VERSIONS => '2147483647', COMPRESSION => 'NONE', TTL => '604800', BLOC
> KSIZE => '8192', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'info', V
> ERSIONS => '10', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '8192',
> IN_MEMORY => 'true', BLOCKCACHE => 'true'}]}}
> .META.,,1 column=info:server, timestamp=1298681408958, value=10.0.1.2:60020
> .META.,,1 column=info:serverstartcode, timestamp=1298681408958, value=1298681408122
>
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
> Sent: Monday, February 28, 2011 2:57 PM
> To: user@hbase.apache.org
> Subject: Re: Question on upgrade chapter
>
> Maybe it's a typo in your email, but it says you need to scan -ROOT-
> to check your .META.'s MEMSTORE_FLUSHSIZE (you seem to imply that you
> scanned .META.)
>
> If MEMSTORE_FLUSHSIZE wasn't changed (default is 64MB) then it won't
> be printed. I'm going to change the documentation to make it clearer.
>
> Finally, I don't know how you got a table with
> MEMSTORE_FLUSHSIZE=16384 but that's probably a very bad performance
> issue.
>
> J-D
>
> On Mon, Feb 28, 2011 at 2:26 PM, Buttler, David <bu...@llnl.gov> wrote:
>> Hi,
>>
>> In
>> http://hbase.apache.org/upgrading.html
>> it says that I should scan my .META. table to check my MEMSTORE_SIZE to ensure that it is not 16MB.
>> It is not at all clear what the correct value should be (e.g., what set_meta_memstore_size.rb will change it to)
>>
>> Also, I do not even see this entry in my table. The closest thing I see is MEMSTORE_FLUSHSIZE=16384.
>> I assume this means that I do not have this particular issue. If that is the case, it might be a nice addendum to say that MEMSTORE_SIZE != MEMSTORE_FLUSHSIZE. And maybe also put the value for 16MB (16777216) in parenthesis to be very clear.
>>
>> Thanks,
>> Dave
>>
>
Zookeeper / Hbase
Posted by Joseph Coleman <jo...@infinitecampus.com>.
Question when setting up the zookeeper master node can I have just
zookeeper on that box or do I need to have Hbase and hadoop installed on
there as well? I am just doing a 3 node setup for an exercise before my
production gear gets here.. So I have a working HDFS cluster 1 master and
one slave just trying to setup Hbase and Zookeeper with The zookeeper
master also serving as the Hbase master and the on one HDFS slave also
being my Hbase slave..