You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Asaf Mesika <as...@gmail.com> on 2012/07/09 20:35:48 UTC
HBase becomes ultra-slaggy
Hi,
My cluster started being incredibly slow in the past 2 days.
I've seen many Blocking updates on the region server logs, which lead me to believe HDFS creates is the bottleneck.
I ran a small test (hadoop fs -copyFromLocal big3_3Giga.tz.gz /tmp) which copies a 3.3G file, and I was surprised to see a rate of 11MB/sec, when it was previously around 28MB/sec.
I have 3 RS computers, hosting DataNode and embedded zookeeper as well.
And 1 Master computer, hosting NameNode as well.
The slowest computer has Core 2 Duo 2.93GHz, 4GB.
What steps can I take to find out the source of this slowdown?
Thanks!
Asaf Mesika
Re: HBase becomes ultra-slaggy
Posted by Asaf Mesika <as...@gmail.com>.
I used it but I didn't like the fact that the graph was updating every
minute and sometimes every 5 minutes.
Sent from my iPhone
On 10 ביול 2012, at 06:57, Otis Gospodnetic <ot...@yahoo.com>
wrote:
Asaf - check the URL in my signature - it's a service for monitoring
various HBase metrics and may visually show you what is going on with
various parts of your 3 servers.
Otis
----
Performance Monitoring for Solr / ElasticSearch / HBase -
http://sematext.com/spm
----- Original Message -----
From: Asaf Mesika <as...@gmail.com>
To: user@hbase.apache.org
Cc:
Sent: Monday, July 9, 2012 2:35 PM
Subject: HBase becomes ultra-slaggy
Hi,
My cluster started being incredibly slow in the past 2 days.
I've seen many Blocking updates on the region server logs, which lead me to
believe HDFS creates is the bottleneck.
I ran a small test (hadoop fs -copyFromLocal big3_3Giga.tz.gz /tmp) which
copies
a 3.3G file, and I was surprised to see a rate of 11MB/sec, when it was
previously around 28MB/sec.
I have 3 RS computers, hosting DataNode and embedded zookeeper as well.
And 1 Master computer, hosting NameNode as well.
The slowest computer has Core 2 Duo 2.93GHz, 4GB.
What steps can I take to find out the source of this slowdown?
Thanks!
Asaf Mesika
Re: HBase becomes ultra-slaggy
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Asaf - check the URL in my signature - it's a service for monitoring various HBase metrics and may visually show you what is going on with various parts of your 3 servers.
Otis
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm
----- Original Message -----
> From: Asaf Mesika <as...@gmail.com>
> To: user@hbase.apache.org
> Cc:
> Sent: Monday, July 9, 2012 2:35 PM
> Subject: HBase becomes ultra-slaggy
>
> Hi,
>
> My cluster started being incredibly slow in the past 2 days.
> I've seen many Blocking updates on the region server logs, which lead me to
> believe HDFS creates is the bottleneck.
> I ran a small test (hadoop fs -copyFromLocal big3_3Giga.tz.gz /tmp) which copies
> a 3.3G file, and I was surprised to see a rate of 11MB/sec, when it was
> previously around 28MB/sec.
>
> I have 3 RS computers, hosting DataNode and embedded zookeeper as well.
> And 1 Master computer, hosting NameNode as well.
>
> The slowest computer has Core 2 Duo 2.93GHz, 4GB.
>
> What steps can I take to find out the source of this slowdown?
>
>
> Thanks!
>
> Asaf Mesika
>
Re: HBase becomes ultra-slaggy
Posted by Asaf Mesika <as...@gmail.com>.
I got JMX counters hooked up to JConsole (couple of them opened).
Do you have any advice from your experience on what metrics I should focus on to spot this issue?
On Jul 9, 2012, at 22:19 PM, Stack wrote:
> On Mon, Jul 9, 2012 at 8:35 PM, Asaf Mesika <as...@gmail.com> wrote:
>> Hi,
>>
>> My cluster started being incredibly slow in the past 2 days.
>> I've seen many Blocking updates on the region server logs, which lead me to believe HDFS creates is the bottleneck.
>> I ran a small test (hadoop fs -copyFromLocal big3_3Giga.tz.gz /tmp) which copies a 3.3G file, and I was surprised to see a rate of 11MB/sec, when it was previously around 28MB/sec.
>>
>> I have 3 RS computers, hosting DataNode and embedded zookeeper as well.
>> And 1 Master computer, hosting NameNode as well.
>>
>> The slowest computer has Core 2 Duo 2.93GHz, 4GB.
>>
>> What steps can I take to find out the source of this slowdown?
>
> Install ganglia or opentsdb and figure what your cluster is up to when its slow.
>
> St.Ack
Re: HBase becomes ultra-slaggy
Posted by Stack <st...@duboce.net>.
On Mon, Jul 9, 2012 at 8:35 PM, Asaf Mesika <as...@gmail.com> wrote:
> Hi,
>
> My cluster started being incredibly slow in the past 2 days.
> I've seen many Blocking updates on the region server logs, which lead me to believe HDFS creates is the bottleneck.
> I ran a small test (hadoop fs -copyFromLocal big3_3Giga.tz.gz /tmp) which copies a 3.3G file, and I was surprised to see a rate of 11MB/sec, when it was previously around 28MB/sec.
>
> I have 3 RS computers, hosting DataNode and embedded zookeeper as well.
> And 1 Master computer, hosting NameNode as well.
>
> The slowest computer has Core 2 Duo 2.93GHz, 4GB.
>
> What steps can I take to find out the source of this slowdown?
Install ganglia or opentsdb and figure what your cluster is up to when its slow.
St.Ack