You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2012/07/07 01:55:33 UTC

[jira] [Created] (HBASE-6346) Metrics updating may be using excessive CPU

Andrew Purtell created HBASE-6346:
-------------------------------------

             Summary: Metrics updating may be using excessive CPU
                 Key: HBASE-6346
                 URL: https://issues.apache.org/jira/browse/HBASE-6346
             Project: HBase
          Issue Type: Bug
         Environment: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
            Reporter: Andrew Purtell
         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png

I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-6346) Observed excessive CPU on "quiescent" cluster

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409820#comment-13409820 ] 

Todd Lipcon commented on HBASE-6346:
------------------------------------

RH6.1+ THP defragmentation perhaps?
                
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
>                 Key: HBASE-6346
>                 URL: https://issues.apache.org/jira/browse/HBASE-6346
>             Project: HBase
>          Issue Type: Bug
>         Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
>            Reporter: Andrew Purtell
>         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-6346) Observed excessive CPU on "quiescent" cluster

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410654#comment-13410654 ] 

Andrew Purtell commented on HBASE-6346:
---------------------------------------

bq. Yeah what's that config Andy? I may have encountered the same problem.

I don't think so. It was a misconfiguration of secure ZooKeeper. Not a problem under local profiling, but chewing CPU up on the cluster.
                
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
>                 Key: HBASE-6346
>                 URL: https://issues.apache.org/jira/browse/HBASE-6346
>             Project: HBase
>          Issue Type: Bug
>         Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
>            Reporter: Andrew Purtell
>         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Comment Edited] (HBASE-6346) Metrics updating may be using excessive CPU

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408463#comment-13408463 ] 

Andrew Purtell edited comment on HBASE-6346 at 7/6/12 11:57 PM:
----------------------------------------------------------------

See attached graph.gif what CPU use on the cluster looks like. Nothing should be running, this is a Hadoop+HBase cluster started up and left alone. 'top' on the hosts shows all CPU time used by the JVM running the regionserver.

Attached as runnable_cpu_profile.png is a snapshot of what jprofiler sees of a 2.0.1+0.94.1 HBase process started in-tree as './bin/hbase master start'. I've also attached a stack dump so you can see it is doing "nothing".

Edit: I haven't missed that most of that time is system time. Wonder what is up with that.
                
      was (Author: apurtell):
    See attached graph.gif what CPU use on the cluster looks like. Nothing should be running, this is a Hadoop+HBase cluster started up and left alone. 'top' on the hosts shows all CPU time used by the JVM running the regionserver.

Attached as runnable_cpu_profile.png is a snapshot of what jprofiler sees of a 2.0.1+0.94.1 HBase process started in-tree as './bin/hbase master start'. I've also attached a stack dump so you can see it is doing "nothing".
                  
> Metrics updating may be using excessive CPU
> -------------------------------------------
>
>                 Key: HBASE-6346
>                 URL: https://issues.apache.org/jira/browse/HBASE-6346
>             Project: HBase
>          Issue Type: Bug
>         Environment: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
>            Reporter: Andrew Purtell
>         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-6346) Observed excessive CPU on "quiescent" cluster

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell updated HBASE-6346:
----------------------------------

    Summary: Observed excessive CPU on "quiescent" cluster  (was: Metrics updating may be using excessive CPU)
    
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
>                 Key: HBASE-6346
>                 URL: https://issues.apache.org/jira/browse/HBASE-6346
>             Project: HBase
>          Issue Type: Bug
>         Environment: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
>            Reporter: Andrew Purtell
>         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-6346) Observed excessive CPU on "quiescent" cluster

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410757#comment-13410757 ] 

Jean-Daniel Cryans commented on HBASE-6346:
-------------------------------------------

That's sad then, means I need to dig into that other issue for 0.94
                
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
>                 Key: HBASE-6346
>                 URL: https://issues.apache.org/jira/browse/HBASE-6346
>             Project: HBase
>          Issue Type: Bug
>         Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
>            Reporter: Andrew Purtell
>         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-6346) Observed excessive CPU on "quiescent" cluster

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell updated HBASE-6346:
----------------------------------

    Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT  (was: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT)
    
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
>                 Key: HBASE-6346
>                 URL: https://issues.apache.org/jira/browse/HBASE-6346
>             Project: HBase
>          Issue Type: Bug
>         Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
>            Reporter: Andrew Purtell
>         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HBASE-6346) Metrics updating may be using excessive CPU

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell updated HBASE-6346:
----------------------------------

    Attachment: jstack.txt
                runnable_cpu_profile.png
                graph.gif

See attached graph.gif. Attached as runnable_cpu_profile.png is a snapshot of what jprofiler sees of a 2.0.1+0.94.1 HBase process started in-tree as './bin/hbase master start'. I've also attached a stack dump so you can see it is doing "nothing".
                
> Metrics updating may be using excessive CPU
> -------------------------------------------
>
>                 Key: HBASE-6346
>                 URL: https://issues.apache.org/jira/browse/HBASE-6346
>             Project: HBase
>          Issue Type: Bug
>         Environment: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
>            Reporter: Andrew Purtell
>         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Resolved] (HBASE-6346) Observed excessive CPU on "quiescent" cluster

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell resolved HBASE-6346.
-----------------------------------

    Resolution: Not A Problem

Aha, got it, sorry for the noise, this is a configuration problem. 
                
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
>                 Key: HBASE-6346
>                 URL: https://issues.apache.org/jira/browse/HBASE-6346
>             Project: HBase
>          Issue Type: Bug
>         Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
>            Reporter: Andrew Purtell
>         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-6346) Observed excessive CPU on "quiescent" cluster

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408466#comment-13408466 ] 

Andrew Purtell commented on HBASE-6346:
---------------------------------------

This is repeatable, any cluster I start shows this behavior. I started seeing this last week, but took various steps first like switching from JRE 7u4 back to 6u33.
                
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
>                 Key: HBASE-6346
>                 URL: https://issues.apache.org/jira/browse/HBASE-6346
>             Project: HBase
>          Issue Type: Bug
>         Environment: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
>            Reporter: Andrew Purtell
>         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-6346) Observed excessive CPU on "quiescent" cluster

Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408472#comment-13408472 ] 

Zhihong Ted Yu commented on HBASE-6346:
---------------------------------------

Can you highlight the configuration change that made high CPU go away ?
                
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
>                 Key: HBASE-6346
>                 URL: https://issues.apache.org/jira/browse/HBASE-6346
>             Project: HBase
>          Issue Type: Bug
>         Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
>            Reporter: Andrew Purtell
>         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Comment Edited] (HBASE-6346) Metrics updating may be using excessive CPU

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408463#comment-13408463 ] 

Andrew Purtell edited comment on HBASE-6346 at 7/6/12 11:55 PM:
----------------------------------------------------------------

See attached graph.gif what CPU use on the cluster looks like. Nothing should be running, this is a Hadoop+HBase cluster started up and left alone. 'top' on the hosts shows all CPU time used by the JVM running the regionserver.

Attached as runnable_cpu_profile.png is a snapshot of what jprofiler sees of a 2.0.1+0.94.1 HBase process started in-tree as './bin/hbase master start'. I've also attached a stack dump so you can see it is doing "nothing".
                
      was (Author: apurtell):
    See attached graph.gif. Attached as runnable_cpu_profile.png is a snapshot of what jprofiler sees of a 2.0.1+0.94.1 HBase process started in-tree as './bin/hbase master start'. I've also attached a stack dump so you can see it is doing "nothing".
                  
> Metrics updating may be using excessive CPU
> -------------------------------------------
>
>                 Key: HBASE-6346
>                 URL: https://issues.apache.org/jira/browse/HBASE-6346
>             Project: HBase
>          Issue Type: Bug
>         Environment: Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
>            Reporter: Andrew Purtell
>         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-6346) Observed excessive CPU on "quiescent" cluster

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409946#comment-13409946 ] 

Jean-Daniel Cryans commented on HBASE-6346:
-------------------------------------------

Yeah what's that config Andy? I may have encountered the same problem.
                
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
>                 Key: HBASE-6346
>                 URL: https://issues.apache.org/jira/browse/HBASE-6346
>             Project: HBase
>          Issue Type: Bug
>         Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
>            Reporter: Andrew Purtell
>         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HBASE-6346) Observed excessive CPU on "quiescent" cluster

Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408468#comment-13408468 ] 

Zhihong Ted Yu commented on HBASE-6346:
---------------------------------------

Looking at RegionServerMetrics, the metrics are collected (pushed) unconditionally.
Can we group them and make the push configurable for individual groups ?
                
> Observed excessive CPU on "quiescent" cluster
> ---------------------------------------------
>
>                 Key: HBASE-6346
>                 URL: https://issues.apache.org/jira/browse/HBASE-6346
>             Project: HBase
>          Issue Type: Bug
>         Environment: Sun JRE 6u33, Hadoop 2.0.1-alpha, HBase 0.94.1-SNAPSHOT
>            Reporter: Andrew Purtell
>         Attachments: graph.gif, jstack.txt, runnable_cpu_profile.png
>
>
> I've only started looking at this but became concerned when a completely quiescent cluster loads up 20% CPU (system+user) once HBase comes up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira