You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ted Tuttle <te...@mentacapital.com> on 2015/03/26 20:18:46 UTC
master consumes large amount of CPU for days
Hello-
Our master process started consuming a large amount of CPU (75% of box) several days back and hasn't stopped. I have 2 questions:
1) what is it doing (stack dump and log below)
2) is it safe to restart the master without taking the whole cluster down?
Master stack dump:
http://pastebin.com/G0iNNEpC
Master log from last 15 mins or so:
http://pastebin.com/WQNjhFGf
Thanks,
Ted
Re: master consumes large amount of CPU for days
Posted by Esteban Gutierrez <es...@cloudera.com>.
Hi Ted,
Since HBase 0.98.8 we do better caching those calls (see HBASE-12219) can
you upgrade to that version? Or if you are using 0.94 and you can't upgrade
to HBase 1 or 0.98 there is a patch in that JIRA for 0.94 that might
improve the situation.
cheers,
esteban.
--
Cloudera, Inc.
On Thu, Mar 26, 2015 at 4:22 PM, Ted Tuttle <te...@mentacapital.com> wrote:
> We have no regions in transition.
>
> As far as who's requesting the table descriptors: We are calling
> HTableIterface.getTableDescriptor frequently. We are going to cache the
> result to avoid the unnecessary network calls.
>
> Thanks for the hint!
>
> From: lars hofhansl [mailto:larsh@apache.org]
> Sent: Thursday, March 26, 2015 3:43 PM
> To: user@hbase.apache.org
> Cc: Development
> Subject: Re: master consumes large amount of CPU for days
>
> Hi Ted,
>
> Yes, it is safe to bounce the HMaster without taking the region servers
> down.
>
> Are there any regions in transition (would be shown on the master's page)?
>
> All of the threads involved with
> -
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(org.apache.hadoop.fs.Path)
> @bci=14, line=309 (Compiled frame)
> ...
> -
> org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(java.util.List)
> @bci=50, line=1932 (Compiled frame)
>
> are a bit suspect. Who's requesting all the table descriptors from the
> master?
> -- Lars
>
> ________________________________
> From: Ted Tuttle <te...@mentacapital.com>>
> To: "user@hbase.apache.org<ma...@hbase.apache.org>" <
> user@hbase.apache.org<ma...@hbase.apache.org>>
> Cc: Development <Development@mentacapital.com<mailto:
> Development@mentacapital.com>>
> Sent: Thursday, March 26, 2015 12:18 PM
> Subject: master consumes large amount of CPU for days
>
> Hello-
>
> Our master process started consuming a large amount of CPU (75% of box)
> several days back and hasn't stopped. I have 2 questions:
>
> 1) what is it doing (stack dump and log below)
> 2) is it safe to restart the master without taking the
> whole cluster down?
>
> Master stack dump:
>
> http://pastebin.com/G0iNNEpC
>
> Master log from last 15 mins or so:
>
> http://pastebin.com/WQNjhFGf
>
> Thanks,
> Ted
>
>
RE: master consumes large amount of CPU for days
Posted by Ted Tuttle <te...@mentacapital.com>.
We have no regions in transition.
As far as who's requesting the table descriptors: We are calling HTableIterface.getTableDescriptor frequently. We are going to cache the result to avoid the unnecessary network calls.
Thanks for the hint!
From: lars hofhansl [mailto:larsh@apache.org]
Sent: Thursday, March 26, 2015 3:43 PM
To: user@hbase.apache.org
Cc: Development
Subject: Re: master consumes large amount of CPU for days
Hi Ted,
Yes, it is safe to bounce the HMaster without taking the region servers down.
Are there any regions in transition (would be shown on the master's page)?
All of the threads involved with
- org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(org.apache.hadoop.fs.Path) @bci=14, line=309 (Compiled frame)
...
- org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(java.util.List) @bci=50, line=1932 (Compiled frame)
are a bit suspect. Who's requesting all the table descriptors from the master?
-- Lars
________________________________
From: Ted Tuttle <te...@mentacapital.com>>
To: "user@hbase.apache.org<ma...@hbase.apache.org>" <us...@hbase.apache.org>>
Cc: Development <De...@mentacapital.com>>
Sent: Thursday, March 26, 2015 12:18 PM
Subject: master consumes large amount of CPU for days
Hello-
Our master process started consuming a large amount of CPU (75% of box) several days back and hasn't stopped. I have 2 questions:
1) what is it doing (stack dump and log below)
2) is it safe to restart the master without taking the whole cluster down?
Master stack dump:
http://pastebin.com/G0iNNEpC
Master log from last 15 mins or so:
http://pastebin.com/WQNjhFGf
Thanks,
Ted
Re: master consumes large amount of CPU for days
Posted by lars hofhansl <la...@apache.org>.
Hi Ted,
Yes, it is safe to bounce the HMaster without taking the region servers down.
Are there any regions in transition (would be shown on the master's page)?
All of the threads involved with
- org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(org.apache.hadoop.fs.Path) @bci=14, line=309 (Compiled frame)
...
- org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(java.util.List) @bci=50, line=1932 (Compiled frame)
are a bit suspect. Who's requesting all the table descriptors from the master?
-- Lars
From: Ted Tuttle <te...@mentacapital.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Cc: Development <De...@mentacapital.com>
Sent: Thursday, March 26, 2015 12:18 PM
Subject: master consumes large amount of CPU for days
Hello-
Our master process started consuming a large amount of CPU (75% of box) several days back and hasn't stopped. I have 2 questions:
1) what is it doing (stack dump and log below)
2) is it safe to restart the master without taking the whole cluster down?
Master stack dump:
http://pastebin.com/G0iNNEpC
Master log from last 15 mins or so:
http://pastebin.com/WQNjhFGf
Thanks,
Ted
Re: master consumes large amount of CPU for days
Posted by Mikhail Antonov <ol...@gmail.com>.
Log only shows that every 5 mins balancers kicks in and finds no work to do.
Could you share a bit more details about the cluster - how much memory
/ cores you have on the nodes, what are JVM properties for hbase
daemons, what version of HBase you're running, what kind of apps you
run on top of it?
Did you observe any master crashes/failovers during this high-CPU period?
Thanks,
Mikhail
On Thu, Mar 26, 2015 at 12:18 PM, Ted Tuttle <te...@mentacapital.com> wrote:
> Hello-
>
> Our master process started consuming a large amount of CPU (75% of box) several days back and hasn't stopped. I have 2 questions:
>
> 1) what is it doing (stack dump and log below)
> 2) is it safe to restart the master without taking the whole cluster down?
>
> Master stack dump:
>
> http://pastebin.com/G0iNNEpC
>
> Master log from last 15 mins or so:
>
> http://pastebin.com/WQNjhFGf
>
> Thanks,
> Ted
>
--
Thanks,
Michael Antonov