You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2012/07/07 01:55:33 UTC
[jira] [Created] (HBASE-6346) Metrics updating may be using
excessive CPU
Andrew Purtell created HBASE-6346:
-------------------------------------
Summary: Metrics updating may be using excessive CPU
Key: HBASE-6346
URL: https://issues.apache.org/jira/browse/HBASE-6346
Project: HBase
Issue Type: Bug
Environment: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
Reporter: Andrew Purtell
Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6346) Observed excessive CPU on
"quiescent" cluster
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409820#comment-13409820 ]
Todd Lipcon commented on HBASE-6346:
------------------------------------
RH6.1+ THP defragmentation perhaps?
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
> Key: HBASE-6346
> URL: https://issues.apache.org/jira/browse/HBASE-6346
> Project: HBase
> Issue Type: Bug
> Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
> Reporter: Andrew Purtell
> Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6346) Observed excessive CPU on
"quiescent" cluster
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410654#comment-13410654 ]
Andrew Purtell commented on HBASE-6346:
---------------------------------------
bq. Yeah what's that config Andy? I may have encountered the same problem.
I don't think so. It was a misconfiguration of secure ZooKeeper. Not a problem under local profiling, but chewing CPU up on the cluster.
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
> Key: HBASE-6346
> URL: https://issues.apache.org/jira/browse/HBASE-6346
> Project: HBase
> Issue Type: Bug
> Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
> Reporter: Andrew Purtell
> Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6346) Metrics updating may be using
excessive CPU
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408463#comment-13408463 ]
Andrew Purtell edited comment on HBASE-6346 at 7/6/12 11:57 PM:
----------------------------------------------------------------
See attached graph.gif what CPU use on the cluster looks like. Nothing should be running, this is a Hadoop+HBase cluster started up and left alone. 'top' on the hosts shows all CPU time used by the JVM running the regionserver.
Attached as runnable_cpu_profile.png is a snapshot of what jprofiler sees of a 2.0.1+0.94.1 HBase process started in-tree as './bin/hbase master start'. I've also attached a stack dump so you can see it is doing "nothing".
Edit: I haven't missed that most of that time is system time. Wonder what is up with that.
was (Author: apurtell):
See attached graph.gif what CPU use on the cluster looks like. Nothing should be running, this is a Hadoop+HBase cluster started up and left alone. 'top' on the hosts shows all CPU time used by the JVM running the regionserver.
Attached as runnable_cpu_profile.png is a snapshot of what jprofiler sees of a 2.0.1+0.94.1 HBase process started in-tree as './bin/hbase master start'. I've also attached a stack dump so you can see it is doing "nothing".
> Metrics updating may be using excessive CPU
> -------------------------------------------
>
> Key: HBASE-6346
> URL: https://issues.apache.org/jira/browse/HBASE-6346
> Project: HBase
> Issue Type: Bug
> Environment: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
> Reporter: Andrew Purtell
> Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6346) Observed excessive CPU on "quiescent"
cluster
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-6346:
----------------------------------
Summary: Observed excessive CPU on "quiescent" cluster (was: Metrics updating may be using excessive CPU)
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
> Key: HBASE-6346
> URL: https://issues.apache.org/jira/browse/HBASE-6346
> Project: HBase
> Issue Type: Bug
> Environment: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
> Reporter: Andrew Purtell
> Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6346) Observed excessive CPU on
"quiescent" cluster
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410757#comment-13410757 ]
Jean-Daniel Cryans commented on HBASE-6346:
-------------------------------------------
That's sad then, means I need to dig into that other issue for 0.94
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
> Key: HBASE-6346
> URL: https://issues.apache.org/jira/browse/HBASE-6346
> Project: HBase
> Issue Type: Bug
> Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
> Reporter: Andrew Purtell
> Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6346) Observed excessive CPU on "quiescent"
cluster
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-6346:
----------------------------------
Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT (was: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT)
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
> Key: HBASE-6346
> URL: https://issues.apache.org/jira/browse/HBASE-6346
> Project: HBase
> Issue Type: Bug
> Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
> Reporter: Andrew Purtell
> Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6346) Metrics updating may be using
excessive CPU
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-6346:
----------------------------------
Attachment: jstack.txt
runnable_cpu_profile.png
graph.gif
See attached graph.gif. Attached as runnable_cpu_profile.png is a snapshot of what jprofiler sees of a 2.0.1+0.94.1 HBase process started in-tree as './bin/hbase master start'. I've also attached a stack dump so you can see it is doing "nothing".
> Metrics updating may be using excessive CPU
> -------------------------------------------
>
> Key: HBASE-6346
> URL: https://issues.apache.org/jira/browse/HBASE-6346
> Project: HBase
> Issue Type: Bug
> Environment: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
> Reporter: Andrew Purtell
> Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6346) Observed excessive CPU on
"quiescent" cluster
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell resolved HBASE-6346.
-----------------------------------
Resolution: Not A Problem
Aha, got it, sorry for the noise, this is a configuration problem.
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
> Key: HBASE-6346
> URL: https://issues.apache.org/jira/browse/HBASE-6346
> Project: HBase
> Issue Type: Bug
> Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
> Reporter: Andrew Purtell
> Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6346) Observed excessive CPU on
"quiescent" cluster
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408466#comment-13408466 ]
Andrew Purtell commented on HBASE-6346:
---------------------------------------
This is repeatable, any cluster I start shows this behavior. I started seeing this last week, but took various steps first like switching from JRE 7u4 back to 6u33.
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
> Key: HBASE-6346
> URL: https://issues.apache.org/jira/browse/HBASE-6346
> Project: HBase
> Issue Type: Bug
> Environment: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
> Reporter: Andrew Purtell
> Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6346) Observed excessive CPU on
"quiescent" cluster
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408472#comment-13408472 ]
Zhihong Ted Yu commented on HBASE-6346:
---------------------------------------
Can you highlight the configuration change that made high CPU go away ?
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
> Key: HBASE-6346
> URL: https://issues.apache.org/jira/browse/HBASE-6346
> Project: HBase
> Issue Type: Bug
> Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
> Reporter: Andrew Purtell
> Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6346) Metrics updating may be using
excessive CPU
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408463#comment-13408463 ]
Andrew Purtell edited comment on HBASE-6346 at 7/6/12 11:55 PM:
----------------------------------------------------------------
See attached graph.gif what CPU use on the cluster looks like. Nothing should be running, this is a Hadoop+HBase cluster started up and left alone. 'top' on the hosts shows all CPU time used by the JVM running the regionserver.
Attached as runnable_cpu_profile.png is a snapshot of what jprofiler sees of a 2.0.1+0.94.1 HBase process started in-tree as './bin/hbase master start'. I've also attached a stack dump so you can see it is doing "nothing".
was (Author: apurtell):
See attached graph.gif. Attached as runnable_cpu_profile.png is a snapshot of what jprofiler sees of a 2.0.1+0.94.1 HBase process started in-tree as './bin/hbase master start'. I've also attached a stack dump so you can see it is doing "nothing".
> Metrics updating may be using excessive CPU
> -------------------------------------------
>
> Key: HBASE-6346
> URL: https://issues.apache.org/jira/browse/HBASE-6346
> Project: HBase
> Issue Type: Bug
> Environment: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
> Reporter: Andrew Purtell
> Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6346) Observed excessive CPU on
"quiescent" cluster
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409946#comment-13409946 ]
Jean-Daniel Cryans commented on HBASE-6346:
-------------------------------------------
Yeah what's that config Andy? I may have encountered the same problem.
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
> Key: HBASE-6346
> URL: https://issues.apache.org/jira/browse/HBASE-6346
> Project: HBase
> Issue Type: Bug
> Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
> Reporter: Andrew Purtell
> Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6346) Observed excessive CPU on
"quiescent" cluster
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408468#comment-13408468 ]
Zhihong Ted Yu commented on HBASE-6346:
---------------------------------------
Looking at RegionServerMetrics, the metrics are collected (pushed) unconditionally.
Can we group them and make the push configurable for individual groups ?
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
> Key: HBASE-6346
> URL: https://issues.apache.org/jira/browse/HBASE-6346
> Project: HBase
> Issue Type: Bug
> Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
> Reporter: Andrew Purtell
> Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira