You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Sandeep Reddy <sa...@outlook.com> on 2016/07/08 06:06:29 UTC
Very High CPU usage in RegionServer
Hi,
We are observing very high CPU load(400 to 600%) in one of our RegionServer to the point where the machine is becoming unresponsive.
At this point the whole cluster of 20+ RegionServers becoming unresponsive.
Before cluster becomes unresponsive we observed following symptoms:
* Huge bandwidth spike
* CPU spikes vertically form normal load to very high usage only in one RegionServer
* Few times even though machine is unresponsive, it sending heartbeats to master
* There is no spike in number of requests to HBase
* We are observed this pattern at least twice is last week
* We don't have any co-processors in any of the region servers
What could be the possible reasons for this kind of behaviour.
We are using hbase-0.98.7, hadoop-2.5.1 versions.
Its production cluster so upgrading to latest version will not be possible right away.
Thanks,
Sandeep.
Thanks,
Sandeep.
Re: Very High CPU usage in RegionServer
Posted by la...@apache.org.
Hi Sandeep,
Can you take a jstack of the RegionServer process when this happens the next time?(DebugDump on the RegionServer UI page, or log into the box and run jstack yourself)
Otherwise we'll be guessing :)
-- Lars
From: Sandeep Reddy <sa...@outlook.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Thursday, July 7, 2016 11:06 PM
Subject: Very High CPU usage in RegionServer
Hi,
We are observing very high CPU load(400 to 600%) in one of our RegionServer to the point where the machine is becoming unresponsive.
At this point the whole cluster of 20+ RegionServers becoming unresponsive.
Before cluster becomes unresponsive we observed following symptoms:
* Huge bandwidth spike
* CPU spikes vertically form normal load to very high usage only in one RegionServer
* Few times even though machine is unresponsive, it sending heartbeats to master
* There is no spike in number of requests to HBase
* We are observed this pattern at least twice is last week
* We don't have any co-processors in any of the region servers
What could be the possible reasons for this kind of behaviour.
We are using hbase-0.98.7, hadoop-2.5.1 versions.
Its production cluster so upgrading to latest version will not be possible right away.
Thanks,
Sandeep.
Thanks,
Sandeep.
Re: Very High CPU usage in RegionServer
Posted by Geovanie Marquez <ge...@gmail.com>.
We had a problem like this once. We localized it to a fuzzy row filter
request. Are you scanning this region with such a filter?
If so, there is a patch out there we applied that got rid of the problem.
On Jul 8, 2016 2:58 AM, "Samir Ahmic" <ah...@gmail.com> wrote:
> Hi Sandeep,
>
> What sort of load is on cluster when this is happening ? What logs say in
> moment when cluster is this state ? It is strange that whole cluster
> is unresponsive i can remember few cases when this is possible:
>
> - "hbase:meta" table is in transition or unavailable
> - zookeeper is overloaded with huge number of connections
> - HDFS isues
>
> Check logs there must be explanation why you have such huge resource usage
> spikes.
>
> Regards
> Samir
>
> On Fri, Jul 8, 2016 at 8:06 AM, Sandeep Reddy <sa...@outlook.com>
> wrote:
>
> > Hi,
> >
> >
> > We are observing very high CPU load(400 to 600%) in one of our
> > RegionServer to the point where the machine is becoming unresponsive.
> >
> > At this point the whole cluster of 20+ RegionServers becoming
> unresponsive.
> >
> > Before cluster becomes unresponsive we observed following symptoms:
> >
> > * Huge bandwidth spike
> > * CPU spikes vertically form normal load to very high usage only in
> > one RegionServer
> > * Few times even though machine is unresponsive, it sending
> heartbeats
> > to master
> > * There is no spike in number of requests to HBase
> > * We are observed this pattern at least twice is last week
> > * We don't have any co-processors in any of the region servers
> >
> > What could be the possible reasons for this kind of behaviour.
> >
> > We are using hbase-0.98.7, hadoop-2.5.1 versions.
> > Its production cluster so upgrading to latest version will not be
> possible
> > right away.
> >
> >
> > Thanks,
> > Sandeep.
> >
> >
> >
> > Thanks,
> > Sandeep.
> >
>
Re: Very High CPU usage in RegionServer
Posted by Samir Ahmic <ah...@gmail.com>.
Hi Sandeep,
What sort of load is on cluster when this is happening ? What logs say in
moment when cluster is this state ? It is strange that whole cluster
is unresponsive i can remember few cases when this is possible:
- "hbase:meta" table is in transition or unavailable
- zookeeper is overloaded with huge number of connections
- HDFS isues
Check logs there must be explanation why you have such huge resource usage
spikes.
Regards
Samir
On Fri, Jul 8, 2016 at 8:06 AM, Sandeep Reddy <sa...@outlook.com>
wrote:
> Hi,
>
>
> We are observing very high CPU load(400 to 600%) in one of our
> RegionServer to the point where the machine is becoming unresponsive.
>
> At this point the whole cluster of 20+ RegionServers becoming unresponsive.
>
> Before cluster becomes unresponsive we observed following symptoms:
>
> * Huge bandwidth spike
> * CPU spikes vertically form normal load to very high usage only in
> one RegionServer
> * Few times even though machine is unresponsive, it sending heartbeats
> to master
> * There is no spike in number of requests to HBase
> * We are observed this pattern at least twice is last week
> * We don't have any co-processors in any of the region servers
>
> What could be the possible reasons for this kind of behaviour.
>
> We are using hbase-0.98.7, hadoop-2.5.1 versions.
> Its production cluster so upgrading to latest version will not be possible
> right away.
>
>
> Thanks,
> Sandeep.
>
>
>
> Thanks,
> Sandeep.
>