HBase high availability status

[Maxim Veksler <ma...@gmail.com>]

Hi everyone,

We've evaluating the (very cool) plethora of new noSQL solutions, just
to name a few: Cassandra, Hive, SimpleDB, Voldemort, Hypertable and of
course HBase.

I'm wondering about the current status of HBase / Hadoop:

How is HBQL?
Do you have good experience with it?
Any rough edges only experience can teach me?

What is the present status about HA of the whole system?
Most blogs[1] mention the requirement for external tools to monitor
the Hadoop Job Tacker?, HBase HBaseMaster ?

Has any of you tried running Hive on top of you HBase data store?
What is your experience with this?

How is the community (this is you :) ?

Just out curiosity - What is the volume of you data?
How are you using the system?
What is your uptime? (availability that is, who cares about single
nodes these days).



Yeah, I know these are indeed a lot of questions. Please keep in mind
that this post, like the rest of content posted on the web discussing
noSQL will probably become hot and will surly be hit by \*some*\
Google searchers...
Would very much appreciate your answers guys.

Thank you,
Maxim.

[1] http://www.cloudera.com/blog/2009/07/hadoop-ha-configuration/

--
Cheers,
Maxim Veksler

"Free as in Freedom" - Do u GNU ?

Re: HBase high availability status

[Ryan Rawson <ry...@gmail.com>]

This JIRA was opened recently:

https://issues.apache.org/jira/browse/HDFS-1051

might be some interesting discussions in there for you.  no code yet,
but there it is.

-ryan

On Thu, Mar 18, 2010 at 11:44 PM, tsuna <ts...@gmail.com> wrote:
> On Thu, Mar 18, 2010 at 11:25 AM, Maxim Veksler <ma...@gmail.com> wrote:
>> What is the present status about HA of the whole system?
>> Most blogs[1] mention the requirement for external tools to monitor
>> the Hadoop Job Tacker?, HBase HBaseMaster ?
>>
>> [1] http://www.cloudera.com/blog/2009/07/hadoop-ha-configuration/
>
> This blog post is for Hadoop.  As far as HBase is concerned, the only
> critical piece of HA that it needs from Hadoop is the NameNode.  HBase
> doesn't even need the Job Tracker, for instance.
>
> The the NameNode is currently the thorn in the foot of Hadoop when it
> comes to HA.  People have come with various solutions to work around
> it, such as:
>  * Using DRDB and LinuxHA to maintain a cold standby (and since it's
> a cold standby it means that failovers are somewhat slow and
> disruptive for your applications).
>  * Having two different HDFS file systems, each with its own
> NameNode, and both share the same pool of DataNodes, so you can
> failover from one to another (but then you can run in consistency
> problems, among other things).
>  * Facebook recently came up with something called AvatarNode
> (http://hadoopblog.blogspot.com/2010/02/hadoop-namenode-high-availability.html)
> to maintain a hot standby.
>  * Some people are just being extra careful with the NameNode machine
> to keep a good uptime.
>
> The solution with the AvatarNode is what seems to me like the only
> potentially viable route among these existing alternatives.  Of course
> it would be better if it was possible to run multiple NameNode
> instances together in a quorum.
>
> Other than that, my understanding is that the HMaster isn't a SPoF to
> HBase.  It's not needed to serve traffic, it just does administrative
> tasks on the cluster (like recovering and redistributing regions when
> a region server crashes) – as far as I know.
>
> The region servers can come and go, HBase should be able to handle
> those disruptions.
>
> The last piece of infrastructure is ZooKeeper, which can be run in a
> quorum of 3, 5 or 7 instances, so it's fairly reliable.
>
> So overall it looks like the only big issue is the HA (or lack
> thereof) situation with the NameNode.
>
> --
> Benoit "tsuna" Sigoure
> Software Engineer @ www.StumbleUpon.com
>

Re: HBase high availability status

[tsuna <ts...@gmail.com>]

On Thu, Mar 18, 2010 at 11:25 AM, Maxim Veksler <ma...@gmail.com> wrote:
> What is the present status about HA of the whole system?
> Most blogs[1] mention the requirement for external tools to monitor
> the Hadoop Job Tacker?, HBase HBaseMaster ?
>
> [1] http://www.cloudera.com/blog/2009/07/hadoop-ha-configuration/

This blog post is for Hadoop.  As far as HBase is concerned, the only
critical piece of HA that it needs from Hadoop is the NameNode.  HBase
doesn't even need the Job Tracker, for instance.

The the NameNode is currently the thorn in the foot of Hadoop when it
comes to HA.  People have come with various solutions to work around
it, such as:
  * Using DRDB and LinuxHA to maintain a cold standby (and since it's
a cold standby it means that failovers are somewhat slow and
disruptive for your applications).
  * Having two different HDFS file systems, each with its own
NameNode, and both share the same pool of DataNodes, so you can
failover from one to another (but then you can run in consistency
problems, among other things).
  * Facebook recently came up with something called AvatarNode
(http://hadoopblog.blogspot.com/2010/02/hadoop-namenode-high-availability.html)
to maintain a hot standby.
  * Some people are just being extra careful with the NameNode machine
to keep a good uptime.

The solution with the AvatarNode is what seems to me like the only
potentially viable route among these existing alternatives.  Of course
it would be better if it was possible to run multiple NameNode
instances together in a quorum.

Other than that, my understanding is that the HMaster isn't a SPoF to
HBase.  It's not needed to serve traffic, it just does administrative
tasks on the cluster (like recovering and redistributing regions when
a region server crashes) – as far as I know.

The region servers can come and go, HBase should be able to handle
those disruptions.

The last piece of infrastructure is ZooKeeper, which can be run in a
quorum of 3, 5 or 7 instances, so it's fairly reliable.

So overall it looks like the only big issue is the HA (or lack
thereof) situation with the NameNode.

-- 
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com