Hbase: GETs are very slow

[Ruben Quintero <rf...@yahoo.com>]

Hi,

I have a hadoop/hbase cluster running on 9 machines (only 8 GB RAM, 1 TB drives), and have recently noticed that Gets from Hbase have slowed down significantly. I'd say at this point I'm not getting more than 100/sec when using the Hbase Java API. DFS-wise, there's plenty of space left (using less than 10%), and all of the servers seem okay. The tables use LZO, and have blockcache disabled (we were having problems inserting up to a billion rows with it on, and read in the mailing list somewhere that it might help). 

The primary table has only 4 million rows at the moment. I created a new test table with only 200,000, and it was running 100/sec as well.

I'm not sure what the problem could be (paging?), or some configuration that can be adjusted?

Any ideas? I can show our configuration if that's helpful, I just wasn't sure what info would be helpful and what would be extraneous. 

Thanks,

- Ruben



      

RE: Hbase: GETs are very slow

[Michael Segel <mi...@hotmail.com>]

But going back to the OP's problem... sounds like there may be a GC issue and changing the timeout on ZK.


> Date: Fri, 30 Apr 2010 16:50:54 -0700
> Subject: Re: Hbase: GETs are very slow
> From: jdcryans@apache.org
> To: hbase-user@hadoop.apache.org
> 
> So we chatted a bit on IRC, the reason GETs were slower is because
> block caching was disabled and all calls were hitting HDFS. I was
> confused by the first email as it seemed that for some time it was
> still speedy without caching.
> 
> I wanted to look at the import issue, but logs weren't available.
> 
> J-D
> 
> On Fri, Apr 30, 2010 at 10:44 AM, Ruben Quintero <rf...@yahoo.com> wrote:
> > We're running 20.3, and it has a 6 GB heap.
> >
> > With block caching on, it seems we were running out of memory.  It would temporarily lose a region server (usually when it attempted to split) and that caused a chain reaction when it attempted to recover.  The heap would start to surge and cause a heavy garbage collection. We would have nodes dropping in and out, and getting overloaded when they rejoined. We found a post in a mailing list that recommended turning off block caching, and it ran well after that.
> >
> > As for swap, that was my first guess. How can I make sure it's not swapping, or is there a way to see if it is?
> >
> > Thanks,
> >
> > - Ruben
> >
> >
> >
> >
> > ________________________________
> > From: Jean-Daniel Cryans <jd...@apache.org>
> > To: hbase-user@hadoop.apache.org
> > Sent: Fri, April 30, 2010 12:27:37 PM
> > Subject: Re: Hbase: GETs are very slow
> >
> > Which version? How much heap was given to HBase?
> >
> > WRT block caching, I don't see how it could impact uploading in any
> > way, you should enable it. What was the problem inserting 1B rows
> > exactly? How were you running the upload?
> >
> > Are you making sure there's no swap on the machines? That kills java
> > performance faster than you can say "hbase" ;)
> >
> > J-D
> >
> > On Fri, Apr 30, 2010 at 8:36 AM, Ruben Quintero <rf...@yahoo.com> wrote:
> >> Hi,
> >>
> >> I have a hadoop/hbase cluster running on 9 machines (only 8 GB RAM, 1 TB drives), and have recently noticed that Gets from Hbase have slowed down significantly. I'd say at this point I'm not getting more than 100/sec when using the Hbase Java API. DFS-wise, there's plenty of space left (using less than 10%), and all of the servers seem okay. The tables use LZO, and have blockcache disabled (we were having problems inserting up to a billion rows with it on, and read in the mailing list somewhere that it might help).
> >>
> >> The primary table has only 4 million rows at the moment. I created a new test table with only 200,000, and it was running 100/sec as well.
> >>
> >> I'm not sure what the problem could be (paging?), or some configuration that can be adjusted?
> >>
> >> Any ideas? I can show our configuration if that's helpful, I just wasn't sure what info would be helpful and what would be extraneous.
> >>
> >> Thanks,
> >>
> >> - Ruben
> >>
> >>
> >>
> >>
> >
> >
> >
> >
 		 	   		  
_________________________________________________________________
Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox.
http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_1

Re: Hbase: GETs are very slow

[Jean-Daniel Cryans <jd...@apache.org>]

So we chatted a bit on IRC, the reason GETs were slower is because
block caching was disabled and all calls were hitting HDFS. I was
confused by the first email as it seemed that for some time it was
still speedy without caching.

I wanted to look at the import issue, but logs weren't available.

J-D

On Fri, Apr 30, 2010 at 10:44 AM, Ruben Quintero <rf...@yahoo.com> wrote:
> We're running 20.3, and it has a 6 GB heap.
>
> With block caching on, it seems we were running out of memory.  It would temporarily lose a region server (usually when it attempted to split) and that caused a chain reaction when it attempted to recover.  The heap would start to surge and cause a heavy garbage collection. We would have nodes dropping in and out, and getting overloaded when they rejoined. We found a post in a mailing list that recommended turning off block caching, and it ran well after that.
>
> As for swap, that was my first guess. How can I make sure it's not swapping, or is there a way to see if it is?
>
> Thanks,
>
> - Ruben
>
>
>
>
> ________________________________
> From: Jean-Daniel Cryans <jd...@apache.org>
> To: hbase-user@hadoop.apache.org
> Sent: Fri, April 30, 2010 12:27:37 PM
> Subject: Re: Hbase: GETs are very slow
>
> Which version? How much heap was given to HBase?
>
> WRT block caching, I don't see how it could impact uploading in any
> way, you should enable it. What was the problem inserting 1B rows
> exactly? How were you running the upload?
>
> Are you making sure there's no swap on the machines? That kills java
> performance faster than you can say "hbase" ;)
>
> J-D
>
> On Fri, Apr 30, 2010 at 8:36 AM, Ruben Quintero <rf...@yahoo.com> wrote:
>> Hi,
>>
>> I have a hadoop/hbase cluster running on 9 machines (only 8 GB RAM, 1 TB drives), and have recently noticed that Gets from Hbase have slowed down significantly. I'd say at this point I'm not getting more than 100/sec when using the Hbase Java API. DFS-wise, there's plenty of space left (using less than 10%), and all of the servers seem okay. The tables use LZO, and have blockcache disabled (we were having problems inserting up to a billion rows with it on, and read in the mailing list somewhere that it might help).
>>
>> The primary table has only 4 million rows at the moment. I created a new test table with only 200,000, and it was running 100/sec as well.
>>
>> I'm not sure what the problem could be (paging?), or some configuration that can be adjusted?
>>
>> Any ideas? I can show our configuration if that's helpful, I just wasn't sure what info would be helpful and what would be extraneous.
>>
>> Thanks,
>>
>> - Ruben
>>
>>
>>
>>
>
>
>
>

Re: Hbase: GETs are very slow

[Ruben Quintero <rf...@yahoo.com>]

We're running 20.3, and it has a 6 GB heap.

With block caching on, it seems we were running out of memory.  It would temporarily lose a region server (usually when it attempted to split) and that caused a chain reaction when it attempted to recover.  The heap would start to surge and cause a heavy garbage collection. We would have nodes dropping in and out, and getting overloaded when they rejoined. We found a post in a mailing list that recommended turning off block caching, and it ran well after that.

As for swap, that was my first guess. How can I make sure it's not swapping, or is there a way to see if it is?

Thanks,

- Ruben




________________________________
From: Jean-Daniel Cryans <jd...@apache.org>
To: hbase-user@hadoop.apache.org
Sent: Fri, April 30, 2010 12:27:37 PM
Subject: Re: Hbase: GETs are very slow

Which version? How much heap was given to HBase?

WRT block caching, I don't see how it could impact uploading in any
way, you should enable it. What was the problem inserting 1B rows
exactly? How were you running the upload?

Are you making sure there's no swap on the machines? That kills java
performance faster than you can say "hbase" ;)

J-D

On Fri, Apr 30, 2010 at 8:36 AM, Ruben Quintero <rf...@yahoo.com> wrote:
> Hi,
>
> I have a hadoop/hbase cluster running on 9 machines (only 8 GB RAM, 1 TB drives), and have recently noticed that Gets from Hbase have slowed down significantly. I'd say at this point I'm not getting more than 100/sec when using the Hbase Java API. DFS-wise, there's plenty of space left (using less than 10%), and all of the servers seem okay. The tables use LZO, and have blockcache disabled (we were having problems inserting up to a billion rows with it on, and read in the mailing list somewhere that it might help).
>
> The primary table has only 4 million rows at the moment. I created a new test table with only 200,000, and it was running 100/sec as well.
>
> I'm not sure what the problem could be (paging?), or some configuration that can be adjusted?
>
> Any ideas? I can show our configuration if that's helpful, I just wasn't sure what info would be helpful and what would be extraneous.
>
> Thanks,
>
> - Ruben
>
>
>
>



      

Re: Hbase: GETs are very slow

[Jean-Daniel Cryans <jd...@apache.org>]

Which version? How much heap was given to HBase?

WRT block caching, I don't see how it could impact uploading in any
way, you should enable it. What was the problem inserting 1B rows
exactly? How were you running the upload?

Are you making sure there's no swap on the machines? That kills java
performance faster than you can say "hbase" ;)

J-D

On Fri, Apr 30, 2010 at 8:36 AM, Ruben Quintero <rf...@yahoo.com> wrote:
> Hi,
>
> I have a hadoop/hbase cluster running on 9 machines (only 8 GB RAM, 1 TB drives), and have recently noticed that Gets from Hbase have slowed down significantly. I'd say at this point I'm not getting more than 100/sec when using the Hbase Java API. DFS-wise, there's plenty of space left (using less than 10%), and all of the servers seem okay. The tables use LZO, and have blockcache disabled (we were having problems inserting up to a billion rows with it on, and read in the mailing list somewhere that it might help).
>
> The primary table has only 4 million rows at the moment. I created a new test table with only 200,000, and it was running 100/sec as well.
>
> I'm not sure what the problem could be (paging?), or some configuration that can be adjusted?
>
> Any ideas? I can show our configuration if that's helpful, I just wasn't sure what info would be helpful and what would be extraneous.
>
> Thanks,
>
> - Ruben
>
>
>
>