HBase is not ready for prime time

[Robert Gonzalez <go...@maxpointinteractive.com>]

We've been using HBase for about a year, consistenly running into problems
where we lost data.  After reading forums and some back and forth with other
Hbase users, we changed our data methodology to save less data per row.
This last time, we upgraded to 0.90 at the recommendation of the hbase
community, cleared off all our data, and started over.  Seemed to be running
ok for a couple of months, until this morning.  One of the regionservers
stopped responding to data requests and we tried to restart it to no avail.
Then we shutdown our processes so that nothing was using HBase and we shut
down HBase and brought it back up.  We waited a little bit, until hbase
status indicated that all the servers were back up.  We turned on our
processes and lo and behold, HBase is broken, getting
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region is not online:
-ROOT-,,0
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2319)
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1607)
    at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
    at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)

And now we can't even shut it down.

Seems that Hbase is just too flaky to depend on for a serious system, we've
not had this type of problem to this degree with conventional DB systems.
Now that we are not saving that much data (we are using large hdfs files for
that) in Hbase, we are probably going to move back to a conventional SQL
system for our control data.


Robert Gonzalez
Maxpoint Interactive

Re: HBase is not ready for prime time

[tsuna <ts...@gmail.com>]

On Tue, Apr 12, 2011 at 9:14 AM, Robert Gonzalez
<go...@maxpointinteractive.com> wrote:
> Seems that Hbase is just too flaky to depend on for a serious system, we've
> not had this type of problem to this degree with conventional DB systems.

I'm sorry to hear that you ran into those issues.  While I agree that
running and operating HBase isn't as simple as it could be, I disagree
with your conclusion.  Other companies are rather successful at
running HBase in production for user-facing low-latency systems, e.g.
StumbleUpon or Facebook.  Scaling things is never easy, whether it's a
"NoSQL" thing or a "MySQL" thing.  But either way, it's doable.

-- 
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com