You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Eran Kutner <er...@gigya.com> on 2010/03/04 11:02:56 UTC
Questions while evaluating HBase
Hi,
I'm evaluating Hbase as a NoSql DB for a large scale, interactive, web
service with very high uptime requirements, and have a few questions to the
community.
1. I assume you've seen this benchmark by Yahoo (
http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf and
http://www.brianfrankcooper.net/pubs/ycsb.pdf). They show three main
problems: latency goes up quite significantly when doing more operations,
operations/sec are capped at about half of the other tested platforms and
adding new nodes interrupts the normal operation of the cluster for a while.
Do you consider these results a problem and if so are there any plans to
address them?
2. While running our tests (most were done using 0.20.2) we've had a few
incidents where a table went into "transition" without ever going out of it.
We had to restart the cluster to release the stuck tables. Is this a common
issue?
3. If I understand correctly then any major upgrade requires completely
shutting down the cluster while doing the upgrade as well as deploying a new
version of the application compiled with the new version client? Did I get
it correctly? Is there any strategy for upgrading while the cluster is still
running?
4. This is more a bug report than a question but it seems that in 0.20.3
the master server doesn't stop cleanly and has to be killed manually. Is
someone else seeing it too?
5. Are there any performance benchmarks for the Thrift gateway? Do you
have an estimate of the performance penalty of using the gateway compared to
using the native API?
6. Right now, my biggest concern about HBase is its administration
complexity and cost. If anyone can share their experience that would be a
huge help. How many serves do you have in the cluster? How much ongoing
effort does it take to administrate it? What uptime levels are you seeing
(including upgrades)? Do you have any good strategy for running one cluster
across two data centers, or replicating between two clusters in two
different DCs? Did you have any serious problems/crashes/downtime with
HBase?
Thanks a lot,
Eran Kutner
Re: Questions while evaluating HBase
Posted by Al Lias <al...@gmx.de>.
Am 04.03.2010 11:02, schrieb Eran Kutner:
> ...
> 6. Right now, my biggest concern about HBase is its administration
> complexity and cost. If anyone can share their experience that would be a
> huge help. How many serves do you have in the cluster? How much ongoing
> effort does it take to administrate it? What uptime levels are you seeing
> (including upgrades)? Do you have any good strategy for running one cluster
> across two data centers, or replicating between two clusters in two
> different DCs? Did you have any serious problems/crashes/downtime with
> HBase?
>
For me, hiring a linux-affine java person and then let him/her learn
HBase is a matter of days, in a few weeks he's getting experienced.
Find a good oracle guy for a bigdata installation and compare ;-)
Al
Re: Questions while evaluating HBase
Posted by Stack <st...@duboce.net>.
On Thu, Mar 4, 2010 at 10:44 AM, Patrick Hunt <ph...@apache.org> wrote:
>> Please see our answer
>>
>> http://www.search-hadoop.com/m?id=7c962aed1002091610q14f2d6f0gc420ddade319fe60@mail.gmail.com
>
> Any eta on when updated results will be available?
>
Not sure. I'm working with Adam tomorrow. Hopefully soon after that.
St.Ack
> Patrick
>
> Jean-Daniel Cryans wrote:
>>
>> Inline.
>>
>> J-D
>>
>>> 1. I assume you've seen this benchmark by Yahoo (
>>> http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf and
>>> http://www.brianfrankcooper.net/pubs/ycsb.pdf). They show three main
>>> problems: latency goes up quite significantly when doing more
>>> operations,
>>> operations/sec are capped at about half of the other tested platforms
>>> and
>>> adding new nodes interrupts the normal operation of the cluster for a
>>> while.
>>> Do you consider these results a problem and if so are there any plans to
>>> address them?
>>
>> Please see our answer
>>
>> http://www.search-hadoop.com/m?id=7c962aed1002091610q14f2d6f0gc420ddade319fe60@mail.gmail.com
>>
>>> 2. While running our tests (most were done using 0.20.2) we've had a few
>>> incidents where a table went into "transition" without ever going out of
>>> it.
>>> We had to restart the cluster to release the stuck tables. Is this a
>>> common
>>> issue?
>>
>> 0.20.3 has a much better story, 0.20.4 will include even more reliability
>> fixes.
>>
>>> 3. If I understand correctly then any major upgrade requires completely
>>> shutting down the cluster while doing the upgrade as well as deploying a
>>> new
>>> version of the application compiled with the new version client? Did I
>>> get
>>> it correctly? Is there any strategy for upgrading while the cluster is
>>> still
>>> running?
>>
>> Lots of different reasons why: Hadoop RPC is versionned, a new Hadoop
>> major version requires filesystem upgrades, etc...
>>
>> So for HBase, you currently can do rolling restarts between minor
>> versions until told otherwise (in the release notes). See
>> http://wiki.apache.org/hadoop/Hbase/RollingRestart
>>
>> Also Hadoop RPC will probably be replaced in the future with Avro and
>> by then all releases should be backward compatible (we hope).
>>
>>> 4. This is more a bug report than a question but it seems that in 0.20.3
>>> the master server doesn't stop cleanly and has to be killed manually. Is
>>> someone else seeing it too?
>>
>> Can you provide more details? Logs and stack traces appreciated.
>>
>>> 5. Are there any performance benchmarks for the Thrift gateway? Do you
>>> have an estimate of the performance penalty of using the gateway
>>> compared to
>>> using the native API?
>>
>> The good thing with thrift servers is that those they have long lived
>> clients so their cache is always full and HotSpot does it's magic. In
>> our tests (we use Thrift servers in production here at StumbleUpon),
>> it's maybe adding 1 or 2 ms per request...
>>
>>> 6. Right now, my biggest concern about HBase is its administration
>>> complexity and cost. If anyone can share their experience that would be
>>> a
>>> huge help. How many serves do you have in the cluster? How much ongoing
>>> effort does it take to administrate it? What uptime levels are you
>>> seeing
>>> (including upgrades)? Do you have any good strategy for running one
>>> cluster
>>> across two data centers, or replicating between two clusters in two
>>> different DCs? Did you have any serious problems/crashes/downtime with
>>> HBase?
>>
>> HBase does require a knowledgeable admin, but which DB doesn't if used
>> on a very large scale? We have a full time DBA here for our mysql
>> clusters but the difference is that those are easier to find than
>> HBase admins, right? So some stats that we can make public:
>>
>> - We have a production cluster, another one for processing and a few
>> other for dev and testing (we have 3 HBase committers on staff so...
>> we need machines!). The production clusters have somewhat beefy nodes,
>> i7s with 24GB of RAM and 4x1TB in JBOD. None has more than 40 nodes.
>>
>> - Cluster replication is actually a feature I'm working on. See
>> http://issues.apache.org/jira/browse/HBASE-1295. We currently have 2
>> clusters replicating to each other, each hosted in a different city
>> and around 50M rows are sent each day (we aren't replicating
>> everything tho).
>>
>> - We did have some good crashes, we even run unofficial releases
>> sometimes, but since we are very knowledgeable we are able to fix
>> those and we always get them committed.
>>
>> - I can't disclose our uptime since it would give hints about uptime
>> of one of our product. I can say tho that it's getting better with
>> every release but eh, HBase is still very bleeding edge.
>>
>>>
>>> Thanks a lot,
>>> Eran Kutner
>>>
>
Re: Questions while evaluating HBase
Posted by Patrick Hunt <ph...@apache.org>.
> Please see our answer
>
http://www.search-hadoop.com/m?id=7c962aed1002091610q14f2d6f0gc420ddade319fe60@mail.gmail.com
Any eta on when updated results will be available?
Patrick
Jean-Daniel Cryans wrote:
> Inline.
>
> J-D
>
>> 1. I assume you've seen this benchmark by Yahoo (
>> http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf and
>> http://www.brianfrankcooper.net/pubs/ycsb.pdf). They show three main
>> problems: latency goes up quite significantly when doing more operations,
>> operations/sec are capped at about half of the other tested platforms and
>> adding new nodes interrupts the normal operation of the cluster for a while.
>> Do you consider these results a problem and if so are there any plans to
>> address them?
>
> Please see our answer
> http://www.search-hadoop.com/m?id=7c962aed1002091610q14f2d6f0gc420ddade319fe60@mail.gmail.com
>
>> 2. While running our tests (most were done using 0.20.2) we've had a few
>> incidents where a table went into "transition" without ever going out of it.
>> We had to restart the cluster to release the stuck tables. Is this a common
>> issue?
>
> 0.20.3 has a much better story, 0.20.4 will include even more reliability fixes.
>
>> 3. If I understand correctly then any major upgrade requires completely
>> shutting down the cluster while doing the upgrade as well as deploying a new
>> version of the application compiled with the new version client? Did I get
>> it correctly? Is there any strategy for upgrading while the cluster is still
>> running?
>
> Lots of different reasons why: Hadoop RPC is versionned, a new Hadoop
> major version requires filesystem upgrades, etc...
>
> So for HBase, you currently can do rolling restarts between minor
> versions until told otherwise (in the release notes). See
> http://wiki.apache.org/hadoop/Hbase/RollingRestart
>
> Also Hadoop RPC will probably be replaced in the future with Avro and
> by then all releases should be backward compatible (we hope).
>
>> 4. This is more a bug report than a question but it seems that in 0.20.3
>> the master server doesn't stop cleanly and has to be killed manually. Is
>> someone else seeing it too?
>
> Can you provide more details? Logs and stack traces appreciated.
>
>> 5. Are there any performance benchmarks for the Thrift gateway? Do you
>> have an estimate of the performance penalty of using the gateway compared to
>> using the native API?
>
> The good thing with thrift servers is that those they have long lived
> clients so their cache is always full and HotSpot does it's magic. In
> our tests (we use Thrift servers in production here at StumbleUpon),
> it's maybe adding 1 or 2 ms per request...
>
>> 6. Right now, my biggest concern about HBase is its administration
>> complexity and cost. If anyone can share their experience that would be a
>> huge help. How many serves do you have in the cluster? How much ongoing
>> effort does it take to administrate it? What uptime levels are you seeing
>> (including upgrades)? Do you have any good strategy for running one cluster
>> across two data centers, or replicating between two clusters in two
>> different DCs? Did you have any serious problems/crashes/downtime with
>> HBase?
>
> HBase does require a knowledgeable admin, but which DB doesn't if used
> on a very large scale? We have a full time DBA here for our mysql
> clusters but the difference is that those are easier to find than
> HBase admins, right? So some stats that we can make public:
>
> - We have a production cluster, another one for processing and a few
> other for dev and testing (we have 3 HBase committers on staff so...
> we need machines!). The production clusters have somewhat beefy nodes,
> i7s with 24GB of RAM and 4x1TB in JBOD. None has more than 40 nodes.
>
> - Cluster replication is actually a feature I'm working on. See
> http://issues.apache.org/jira/browse/HBASE-1295. We currently have 2
> clusters replicating to each other, each hosted in a different city
> and around 50M rows are sent each day (we aren't replicating
> everything tho).
>
> - We did have some good crashes, we even run unofficial releases
> sometimes, but since we are very knowledgeable we are able to fix
> those and we always get them committed.
>
> - I can't disclose our uptime since it would give hints about uptime
> of one of our product. I can say tho that it's getting better with
> every release but eh, HBase is still very bleeding edge.
>
>>
>> Thanks a lot,
>> Eran Kutner
>>
Re: Questions while evaluating HBase
Posted by Stack <st...@duboce.net>.
On Fri, Mar 5, 2010 at 3:08 AM, Eran Kutner <er...@gigya-inc.com> wrote:
>>
> My understanding is that 0.21 doesn't have an expected release date yet. Right?
>
Correct.
>> > As for the issue with shutting down the master node, what I see is that
>> > running "hbase-daemon.sh stop master" continues printing dots forever.
>> > Looking at the code for that script, it is trying to run "./hbase master
>> > stop". If I run that command manually it seems to ignore the stop parameter
>> > and trying to load another instance of the server which fails in my case
>> > because the server is already running and the JMX port is busy. There is
>> > nothing in the log and the out file only has the exception thrown by the JMX
>> > trying to bind to the busy socket.
>>
>> That is the problem. That bind happens when the JVM boots and makes it
>> exit, so it really doesn't stop the master at all. 0.20.3 has a fix
>> for that.
>
> I am already using 0.20.3
The way shutdown works is that an instance of master sends message to
running master telling it to shutdown. The JMX port clash seems to be
in the way of this happening. I'm sure there is a better way, but
mayhaps for now set alternate JMX options by doing something like:
HBASE_MASTER_OPTS=" -Dcom.sun.management.jmxremote.port=10102"
./bin/stop-hbase.sh
..i.e. a JMX port other than the one being used by the currently
running master (See bin/hbase-env.sh for more info on these options)
St.Ack
Re: Questions while evaluating HBase
Posted by Eran Kutner <er...@gigya-inc.com>.
>
> Well we go through most of the problems, unless you use HBase in ways
> very different from ours (like 10x the machine and load), then you
> might hit issues. But, we are very helpful and usually fix issues in a
> very timely fashion. We are also always available in our IRC channel,
> #hbase on freenode.
I know that I've asked a few questions in IRC and you or someone else
usually answered in a few minutes. This may be a good opportunity to
thank you again for that :)
>
> It will be in 0.21, it's not production ready (we are using it on our
> processing cluster which, as you can guess, runs on hadoop 0.21 and
> hbase trunk... both unreleased). For 0.20... we'll see, depends on
> other stuff going on currently. The cyclic stuff still misses some
> parts (not that bad), selective replication is done on the family
> level and it's currently working.
My understanding is that 0.21 doesn't have an expected release date yet. Right?
>
> >
> > As for the issue with shutting down the master node, what I see is that
> > running "hbase-daemon.sh stop master" continues printing dots forever.
> > Looking at the code for that script, it is trying to run "./hbase master
> > stop". If I run that command manually it seems to ignore the stop parameter
> > and trying to load another instance of the server which fails in my case
> > because the server is already running and the JMX port is busy. There is
> > nothing in the log and the out file only has the exception thrown by the JMX
> > trying to bind to the busy socket.
>
> That is the problem. That bind happens when the JVM boots and makes it
> exit, so it really doesn't stop the master at all. 0.20.3 has a fix
> for that.
I am already using 0.20.3
-eran
Re: Questions while evaluating HBase
Posted by Jean-Daniel Cryans <jd...@apache.org>.
> Unfortunately that is also our achilles' heel, we are far from being Java
> experts and it will probably take us a lot of time to become experts so we
> can debug and fix problems like you do.
Well we go through most of the problems, unless you use HBase in ways
very different from ours (like 10x the machine and load), then you
might hit issues. But, we are very helpful and usually fix issues in a
very timely fashion. We are also always available in our IRC channel,
#hbase on freenode.
> My thinking was to build two independent clusters with cyclic replication so if one crashes we can switch
> to the other one while we figure out how to fix the first. However, doing
> that requires solid replication capabilities. Can I understand from your
> description that you have cyclic, selective replication working in
> production already? I see that it's scheduled to be released on 0.21, is it
> possible to get it to work on 0.20?
It will be in 0.21, it's not production ready (we are using it on our
processing cluster which, as you can guess, runs on hadoop 0.21 and
hbase trunk... both unreleased). For 0.20... we'll see, depends on
other stuff going on currently. The cyclic stuff still misses some
parts (not that bad), selective replication is done on the family
level and it's currently working.
>
> As for the issue with shutting down the master node, what I see is that
> running "hbase-daemon.sh stop master" continues printing dots forever.
> Looking at the code for that script, it is trying to run "./hbase master
> stop". If I run that command manually it seems to ignore the stop parameter
> and trying to load another instance of the server which fails in my case
> because the server is already running and the JMX port is busy. There is
> nothing in the log and the out file only has the exception thrown by the JMX
> trying to bind to the busy socket.
That is the problem. That bind happens when the JVM boots and makes it
exit, so it really doesn't stop the master at all. 0.20.3 has a fix
for that.
>
> Thanks again, I really appreciate the information.
>
> -eran
Re: Questions while evaluating HBase
Posted by Eran Kutner <er...@gigya-inc.com>.
Thanks J-D, that's very helpful.
Your information about Thrift is great. Our primary development language is
C# so using Thrift will allow us to connect our existing code with HBase and
the penalty seems low enough to be worth it.
Unfortunately that is also our achilles' heel, we are far from being Java
experts and it will probably take us a lot of time to become experts so we
can debug and fix problems like you do. My thinking was to build two
independent clusters with cyclic replication so if one crashes we can switch
to the other one while we figure out how to fix the first. However, doing
that requires solid replication capabilities. Can I understand from your
description that you have cyclic, selective replication working in
production already? I see that it's scheduled to be released on 0.21, is it
possible to get it to work on 0.20?
As for the issue with shutting down the master node, what I see is that
running "hbase-daemon.sh stop master" continues printing dots forever.
Looking at the code for that script, it is trying to run "./hbase master
stop". If I run that command manually it seems to ignore the stop parameter
and trying to load another instance of the server which fails in my case
because the server is already running and the JMX port is busy. There is
nothing in the log and the out file only has the exception thrown by the JMX
trying to bind to the busy socket.
Thanks again, I really appreciate the information.
-eran
On Thu, Mar 4, 2010 at 20:28, Jean-Daniel Cryans <jd...@apache.org>wrote:
> Inline.
>
> J-D
>
> > 1. I assume you've seen this benchmark by Yahoo (
> > http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf and
> > http://www.brianfrankcooper.net/pubs/ycsb.pdf). They show three main
> > problems: latency goes up quite significantly when doing more
> operations,
> > operations/sec are capped at about half of the other tested platforms
> and
> > adding new nodes interrupts the normal operation of the cluster for a
> while.
> > Do you consider these results a problem and if so are there any plans
> to
> > address them?
>
> Please see our answer
>
> http://www.search-hadoop.com/m?id=7c962aed1002091610q14f2d6f0gc420ddade319fe60@mail.gmail.com
>
> > 2. While running our tests (most were done using 0.20.2) we've had a
> few
> > incidents where a table went into "transition" without ever going out
> of it.
> > We had to restart the cluster to release the stuck tables. Is this a
> common
> > issue?
>
> 0.20.3 has a much better story, 0.20.4 will include even more reliability
> fixes.
>
> > 3. If I understand correctly then any major upgrade requires completely
> > shutting down the cluster while doing the upgrade as well as deploying
> a new
> > version of the application compiled with the new version client? Did I
> get
> > it correctly? Is there any strategy for upgrading while the cluster is
> still
> > running?
>
> Lots of different reasons why: Hadoop RPC is versionned, a new Hadoop
> major version requires filesystem upgrades, etc...
>
> So for HBase, you currently can do rolling restarts between minor
> versions until told otherwise (in the release notes). See
> http://wiki.apache.org/hadoop/Hbase/RollingRestart
>
> Also Hadoop RPC will probably be replaced in the future with Avro and
> by then all releases should be backward compatible (we hope).
>
> > 4. This is more a bug report than a question but it seems that in
> 0.20.3
> > the master server doesn't stop cleanly and has to be killed manually.
> Is
> > someone else seeing it too?
>
> Can you provide more details? Logs and stack traces appreciated.
>
> > 5. Are there any performance benchmarks for the Thrift gateway? Do you
> > have an estimate of the performance penalty of using the gateway
> compared to
> > using the native API?
>
> The good thing with thrift servers is that those they have long lived
> clients so their cache is always full and HotSpot does it's magic. In
> our tests (we use Thrift servers in production here at StumbleUpon),
> it's maybe adding 1 or 2 ms per request...
>
> > 6. Right now, my biggest concern about HBase is its administration
> > complexity and cost. If anyone can share their experience that would be
> a
> > huge help. How many serves do you have in the cluster? How much ongoing
> > effort does it take to administrate it? What uptime levels are you
> seeing
> > (including upgrades)? Do you have any good strategy for running one
> cluster
> > across two data centers, or replicating between two clusters in two
> > different DCs? Did you have any serious problems/crashes/downtime with
> > HBase?
>
> HBase does require a knowledgeable admin, but which DB doesn't if used
> on a very large scale? We have a full time DBA here for our mysql
> clusters but the difference is that those are easier to find than
> HBase admins, right? So some stats that we can make public:
>
> - We have a production cluster, another one for processing and a few
> other for dev and testing (we have 3 HBase committers on staff so...
> we need machines!). The production clusters have somewhat beefy nodes,
> i7s with 24GB of RAM and 4x1TB in JBOD. None has more than 40 nodes.
>
> - Cluster replication is actually a feature I'm working on. See
> http://issues.apache.org/jira/browse/HBASE-1295. We currently have 2
> clusters replicating to each other, each hosted in a different city
> and around 50M rows are sent each day (we aren't replicating
> everything tho).
>
> - We did have some good crashes, we even run unofficial releases
> sometimes, but since we are very knowledgeable we are able to fix
> those and we always get them committed.
>
> - I can't disclose our uptime since it would give hints about uptime
> of one of our product. I can say tho that it's getting better with
> every release but eh, HBase is still very bleeding edge.
>
> >
> >
> > Thanks a lot,
> > Eran Kutner
> >
>
Re: Questions while evaluating HBase
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Inline.
J-D
> 1. I assume you've seen this benchmark by Yahoo (
> http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf and
> http://www.brianfrankcooper.net/pubs/ycsb.pdf). They show three main
> problems: latency goes up quite significantly when doing more operations,
> operations/sec are capped at about half of the other tested platforms and
> adding new nodes interrupts the normal operation of the cluster for a while.
> Do you consider these results a problem and if so are there any plans to
> address them?
Please see our answer
http://www.search-hadoop.com/m?id=7c962aed1002091610q14f2d6f0gc420ddade319fe60@mail.gmail.com
> 2. While running our tests (most were done using 0.20.2) we've had a few
> incidents where a table went into "transition" without ever going out of it.
> We had to restart the cluster to release the stuck tables. Is this a common
> issue?
0.20.3 has a much better story, 0.20.4 will include even more reliability fixes.
> 3. If I understand correctly then any major upgrade requires completely
> shutting down the cluster while doing the upgrade as well as deploying a new
> version of the application compiled with the new version client? Did I get
> it correctly? Is there any strategy for upgrading while the cluster is still
> running?
Lots of different reasons why: Hadoop RPC is versionned, a new Hadoop
major version requires filesystem upgrades, etc...
So for HBase, you currently can do rolling restarts between minor
versions until told otherwise (in the release notes). See
http://wiki.apache.org/hadoop/Hbase/RollingRestart
Also Hadoop RPC will probably be replaced in the future with Avro and
by then all releases should be backward compatible (we hope).
> 4. This is more a bug report than a question but it seems that in 0.20.3
> the master server doesn't stop cleanly and has to be killed manually. Is
> someone else seeing it too?
Can you provide more details? Logs and stack traces appreciated.
> 5. Are there any performance benchmarks for the Thrift gateway? Do you
> have an estimate of the performance penalty of using the gateway compared to
> using the native API?
The good thing with thrift servers is that those they have long lived
clients so their cache is always full and HotSpot does it's magic. In
our tests (we use Thrift servers in production here at StumbleUpon),
it's maybe adding 1 or 2 ms per request...
> 6. Right now, my biggest concern about HBase is its administration
> complexity and cost. If anyone can share their experience that would be a
> huge help. How many serves do you have in the cluster? How much ongoing
> effort does it take to administrate it? What uptime levels are you seeing
> (including upgrades)? Do you have any good strategy for running one cluster
> across two data centers, or replicating between two clusters in two
> different DCs? Did you have any serious problems/crashes/downtime with
> HBase?
HBase does require a knowledgeable admin, but which DB doesn't if used
on a very large scale? We have a full time DBA here for our mysql
clusters but the difference is that those are easier to find than
HBase admins, right? So some stats that we can make public:
- We have a production cluster, another one for processing and a few
other for dev and testing (we have 3 HBase committers on staff so...
we need machines!). The production clusters have somewhat beefy nodes,
i7s with 24GB of RAM and 4x1TB in JBOD. None has more than 40 nodes.
- Cluster replication is actually a feature I'm working on. See
http://issues.apache.org/jira/browse/HBASE-1295. We currently have 2
clusters replicating to each other, each hosted in a different city
and around 50M rows are sent each day (we aren't replicating
everything tho).
- We did have some good crashes, we even run unofficial releases
sometimes, but since we are very knowledgeable we are able to fix
those and we always get them committed.
- I can't disclose our uptime since it would give hints about uptime
of one of our product. I can say tho that it's getting better with
every release but eh, HBase is still very bleeding edge.
>
>
> Thanks a lot,
> Eran Kutner
>
Re: Questions while evaluating HBase
Posted by Stack <st...@duboce.net>.
On Thu, Mar 4, 2010 at 7:08 PM, Hua Su <hu...@gmail.com> wrote:
> On Fri, Mar 5, 2010 at 2:34 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
>
> I've another question about thrift api. In 20.3, the thrift api lacks of
> many functions which native java api has, such as scanner with time stamp
> range, scanner with server side fileter, etc..
>
> Is there a patch to make thrift api as a full function interface as the
> native java api?
>
Good progress is being made over in
https://issues.apache.org/jira/browse/HBASE-1744 (and in other
issues).
> And, when hadoop's RPC will be replaced with Avro? Before that happens,
> thrift api is I think the best choice for languages other than Java.
>
Yes.
On when Avro goes into hadoop, the avronauts are making noises about
adding it to hadoop trunk soon so should be part of next hadoop
release.
St.Ack
Re: Questions while evaluating HBase
Posted by Hua Su <hu...@gmail.com>.
On Fri, Mar 5, 2010 at 2:34 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> > I also have concern about thrift gateway performance.
> >
> > Maybe I misunderstand but after I have a glance at the code of hbase
> thrift
> > stuff, I find out that there is only one thrift server running. All
> requests
> > are sent to the thrift server first and then forwarded to region servers.
> >
> > Is this a single failure point and a potential performance bottle neck?
>
>
> You can run as many server as you want, we actually run 1 per region
> server in our production environment. Instead of running
> ./bin/hbase-daemon.sh start thrift, do the following to have 1 per RS:
> ./bin/hbase-daemons.sh start thrift
>
Thank you for pointing me out the solution.
I've another question about thrift api. In 20.3, the thrift api lacks of
many functions which native java api has, such as scanner with time stamp
range, scanner with server side fileter, etc..
Is there a patch to make thrift api as a full function interface as the
native java api?
And, when hadoop's RPC will be replaced with Avro? Before that happens,
thrift api is I think the best choice for languages other than Java.
>
> It's really just a wrapper around the java client so it scales to how
> many nodes you have.
>
> J-D
>
Thanks,
- Hua
Re: Questions while evaluating HBase
Posted by Jean-Daniel Cryans <jd...@apache.org>.
> I also have concern about thrift gateway performance.
>
> Maybe I misunderstand but after I have a glance at the code of hbase thrift
> stuff, I find out that there is only one thrift server running. All requests
> are sent to the thrift server first and then forwarded to region servers.
>
> Is this a single failure point and a potential performance bottle neck?
You can run as many server as you want, we actually run 1 per region
server in our production environment. Instead of running
./bin/hbase-daemon.sh start thrift, do the following to have 1 per RS:
./bin/hbase-daemons.sh start thrift
It's really just a wrapper around the java client so it scales to how
many nodes you have.
J-D
Re: Questions while evaluating HBase
Posted by Hua Su <hu...@gmail.com>.
On Thu, Mar 4, 2010 at 6:02 PM, Eran Kutner <er...@gigya.com> wrote:
> Hi,
> I'm evaluating Hbase as a NoSql DB for a large scale, interactive, web
> service with very high uptime requirements, and have a few questions to the
> community.
>
>
> 1. I assume you've seen this benchmark by Yahoo (
> http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf and
> http://www.brianfrankcooper.net/pubs/ycsb.pdf). They show three main
> problems: latency goes up quite significantly when doing more operations,
> operations/sec are capped at about half of the other tested platforms and
> adding new nodes interrupts the normal operation of the cluster for a
> while.
> Do you consider these results a problem and if so are there any plans to
> address them?
> 2. While running our tests (most were done using 0.20.2) we've had a few
> incidents where a table went into "transition" without ever going out of
> it.
> We had to restart the cluster to release the stuck tables. Is this a
> common
> issue?
> 3. If I understand correctly then any major upgrade requires completely
> shutting down the cluster while doing the upgrade as well as deploying a
> new
> version of the application compiled with the new version client? Did I
> get
> it correctly? Is there any strategy for upgrading while the cluster is
> still
> running?
> 4. This is more a bug report than a question but it seems that in 0.20.3
> the master server doesn't stop cleanly and has to be killed manually. Is
> someone else seeing it too?
> 5. Are there any performance benchmarks for the Thrift gateway? Do you
> have an estimate of the performance penalty of using the gateway compared
> to
> using the native API?
>
I also have concern about thrift gateway performance.
Maybe I misunderstand but after I have a glance at the code of hbase thrift
stuff, I find out that there is only one thrift server running. All requests
are sent to the thrift server first and then forwarded to region servers.
Is this a single failure point and a potential performance bottle neck?
> 6. Right now, my biggest concern about HBase is its administration
> complexity and cost. If anyone can share their experience that would be a
> huge help. How many serves do you have in the cluster? How much ongoing
> effort does it take to administrate it? What uptime levels are you seeing
> (including upgrades)? Do you have any good strategy for running one
> cluster
> across two data centers, or replicating between two clusters in two
> different DCs? Did you have any serious problems/crashes/downtime with
> HBase?
>
>
> Thanks a lot,
> Eran Kutner
>
Best,
- Hua