You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Yossi Ittach <yo...@gmail.com> on 2008/10/27 09:32:58 UTC

HBase keeps crushing after load

Hi All

I've been benchmarking HBase for quite sometime now , and it keeps crushing
under high load.

The benchmark scheme is running on a dedicated host (320GB disk , 4 GB ram ,
2 dualcore processors) that runs VMware with 5 virtual stations:
1 - Hadoop NameNode
2 - Hbase Master
3 - Hadoop DataNode
4 - Hbase RegionServer
5 - Hbase RegionServer

The benchmark is a program that loads the HBase with ~ 3.5M files , with avg
size of 5k - total of about 30GB.
HBase has 1 table: *row*=file name , column-family (with just one column) *
column* = content (the file content)

When I'm running a smaller benchmark (350K files ~ 3 GB) it works fine. When
I run the full benchmark - hell breaks loose. After ~1M files , the
regionservers start to report connection problems (region server can't
connect to master , master can't connect to region server , ...) and/or shut
themselves down one after another.

I figured it may be some overload on the host server , but the resources are
fine - a lot of cpu , memory and disk are available (even on peak times) .

I attache part of the log files , if you find it helpful.

Thank you very much!

Vale et me ama
Yossi

Re: HBase keeps crushing after load

Posted by Yossi Ittach <yo...@gmail.com>.
Thank you very much - This indeed was the problem. I hijacked some company
resources and it works now :)


Vale et me ama
Yossi


On Mon, Oct 27, 2008 at 5:48 PM, Jonathan Gray <jl...@streamy.com> wrote:

> That kind of load on 5 virtual machines with only 2 cores and 4GB is not
> realistic.
>
> Typically you'd want a few independent machines with those kinds of specs.
>
> You're probably seeing thread starvation (only 2 cores but each
> regionserver
> is likely maxing one out) as well as swapping.  When that happens, all hell
> breaks loose.
>
> If you're trying to experiment and benchmark before you have the hardware
> for your cluster, you might try XLarge instances on Amazon EC2.
>
> JG
>
> From: Yossi Ittach [mailto:yossale@gmail.com]
> Sent: Monday, October 27, 2008 1:33 AM
> To: hbase-user@hadoop.apache.org
> Subject: HBase keeps crushing after load
>
> Hi All
>
> I've been benchmarking HBase for quite sometime now , and it keeps crushing
> under high load.
>
> The benchmark scheme is running on a dedicated host (320GB disk , 4 GB ram
> ,
> 2 dualcore processors) that runs VMware with 5 virtual stations:
> 1 - Hadoop NameNode
> 2 - Hbase Master
> 3 - Hadoop DataNode
> 4 - Hbase RegionServer
> 5 - Hbase RegionServer
>
> The benchmark is a program that loads the HBase with ~ 3.5M files , with
> avg
> size of 5k - total of about 30GB.
> HBase has 1 table: row=file name , column-family (with just one column)
> column = content (the file content)
>
> When I'm running a smaller benchmark (350K files ~ 3 GB) it works fine.
> When
> I run the full benchmark - hell breaks loose. After ~1M files , the
> regionservers start to report connection problems (region server can't
> connect to master , master can't connect to region server , ...) and/or
> shut
> themselves down one after another.
>
> I figured it may be some overload on the host server , but the resources
> are
> fine - a lot of cpu , memory and disk are available (even on peak times) .
>
> I attache part of the log files , if you find it helpful.
>
> Thank you very much!
>
> Vale et me ama
> Yossi
>
>

RE: HBase keeps crushing after load

Posted by Jonathan Gray <jl...@streamy.com>.
That kind of load on 5 virtual machines with only 2 cores and 4GB is not
realistic.

Typically you’d want a few independent machines with those kinds of specs.

You're probably seeing thread starvation (only 2 cores but each regionserver
is likely maxing one out) as well as swapping.  When that happens, all hell
breaks loose.

If you're trying to experiment and benchmark before you have the hardware
for your cluster, you might try XLarge instances on Amazon EC2.

JG

From: Yossi Ittach [mailto:yossale@gmail.com] 
Sent: Monday, October 27, 2008 1:33 AM
To: hbase-user@hadoop.apache.org
Subject: HBase keeps crushing after load

Hi All

I've been benchmarking HBase for quite sometime now , and it keeps crushing
under high load. 

The benchmark scheme is running on a dedicated host (320GB disk , 4 GB ram ,
2 dualcore processors) that runs VMware with 5 virtual stations:
1 - Hadoop NameNode 
2 - Hbase Master
3 - Hadoop DataNode
4 - Hbase RegionServer
5 - Hbase RegionServer

The benchmark is a program that loads the HBase with ~ 3.5M files , with avg
size of 5k - total of about 30GB.  
HBase has 1 table: row=file name , column-family (with just one column)
column = content (the file content)

When I'm running a smaller benchmark (350K files ~ 3 GB) it works fine. When
I run the full benchmark - hell breaks loose. After ~1M files , the
regionservers start to report connection problems (region server can't
connect to master , master can't connect to region server , ...) and/or shut
themselves down one after another. 

I figured it may be some overload on the host server , but the resources are
fine - a lot of cpu , memory and disk are available (even on peak times) . 

I attache part of the log files , if you find it helpful. 

Thank you very much!

Vale et me ama
Yossi