You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Elliott Clark (JIRA)" <ji...@apache.org> on 2012/08/17 01:17:37 UTC
[jira] [Created] (HBASE-6602) Redion Server Dynamic Metrics can
cause high cpu usage.
Elliott Clark created HBASE-6602:
------------------------------------
Summary: Redion Server Dynamic Metrics can cause high cpu usage.
Key: HBASE-6602
URL: https://issues.apache.org/jira/browse/HBASE-6602
Project: HBase
Issue Type: Bug
Reporter: Elliott Clark
When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
Seeing jstacks like this:
"RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.put(HashMap.java:374)
at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
--
at java.util.TimerThread.run(Timer.java:462)
"Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.put(HashMap.java:374)
at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "Elliott Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-6602:
---------------------------------
Summary: Region Server Dynamic Metrics can cause high cpu usage. (was: Redion Server Dynamic Metrics can cause high cpu usage.)
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Attachments: HBASE-6602-0.94-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436423#comment-13436423 ]
Zhihong Ted Yu commented on HBASE-6602:
---------------------------------------
Patch looks good.
Maybe renaming needsUpdateMessage as metricsNeedUpdate.
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Attachments: HBASE-6602-0.94-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-6602:
-------------------------
Status: Patch Available (was: Open)
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6602-0.94-0.patch, HBASE-6602-trunk-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436925#comment-13436925 ]
Hadoop QA commented on HBASE-6602:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12541386/HBASE-6602-trunk-0.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings).
-1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2605//console
This message is automatically generated.
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6602-0.94-0.patch, HBASE-6602-trunk-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-6602:
-------------------------
Priority: Critical (was: Major)
Fix Version/s: 0.94.2
0.94.2 and critical because can end up burning loads of CPU
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6602-0.94-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6602) Redion Server Dynamic Metrics can
cause high cpu usage.
Posted by "Elliott Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Elliott Clark updated HBASE-6602:
---------------------------------
Attachment: HBASE-6602-0.94-0.patch
Attaching the 0.94 branch patch as hopefully we will move fully over to metrics2 and this issue will go away.
> Redion Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Attachments: HBASE-6602-0.94-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "Zhihong Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436464#comment-13436464 ]
Zhihong Ted Yu commented on HBASE-6602:
---------------------------------------
I am fine with the current naming.
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6602-0.94-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448297#comment-13448297 ]
Hudson commented on HBASE-6602:
-------------------------------
Integrated in HBase-0.94-security-on-Hadoop-23 #7 (See [https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/7/])
HBASE-6602 Region Server Dynamic Metrics can cause high cpu usage. (Revision 1374398)
Result = FAILURE
stack :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Assignee: Elliott Clark
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6602-0.94-0.patch, HBASE-6602-trunk-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439508#comment-13439508 ]
Hudson commented on HBASE-6602:
-------------------------------
Integrated in HBase-0.94-security #48 (See [https://builds.apache.org/job/HBase-0.94-security/48/])
HBASE-6602 Region Server Dynamic Metrics can cause high cpu usage. (Revision 1374398)
Result = FAILURE
stack :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Assignee: Elliott Clark
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6602-0.94-0.patch, HBASE-6602-trunk-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437031#comment-13437031 ]
Hudson commented on HBASE-6602:
-------------------------------
Integrated in HBase-0.94 #406 (See [https://builds.apache.org/job/HBase-0.94/406/])
HBASE-6602 Region Server Dynamic Metrics can cause high cpu usage. (Revision 1374398)
Result = FAILURE
stack :
Files :
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Assignee: Elliott Clark
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6602-0.94-0.patch, HBASE-6602-trunk-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "Elliott Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436448#comment-13436448 ]
Elliott Clark commented on HBASE-6602:
--------------------------------------
The term message implies the async nature of function call so I prefer needsUpdateMessage; I can change it if you're stuck on that.
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Attachments: HBASE-6602-0.94-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436457#comment-13436457 ]
stack commented on HBASE-6602:
------------------------------
+1
I asked Elliott if flag needs to be a data member rather than local variable and he thought about having methods return true/false if needs update... but turns out the interaction w/ clear requires that it be a data member.
needsUpdateMessage seems fine to me.
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Attachments: HBASE-6602-0.94-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437018#comment-13437018 ]
Hudson commented on HBASE-6602:
-------------------------------
Integrated in HBase-TRUNK #3233 (See [https://builds.apache.org/job/HBase-TRUNK/3233/])
HBASE-6602 Region Server Dynamic Metrics can cause high cpu usage. (Revision 1374397)
Result = SUCCESS
stack :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Assignee: Elliott Clark
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6602-0.94-0.patch, HBASE-6602-trunk-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-6602:
-------------------------
Attachment: HBASE-6602-trunk-0.patch
Trunk version.
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6602-0.94-0.patch, HBASE-6602-trunk-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6602) Region Server Dynamic Metrics can
cause high cpu usage.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-6602:
-------------------------
Resolution: Fixed
Assignee: Elliott Clark
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to 0.94 and trunk. Thanks for the patch Mr. Elliott.
> Region Server Dynamic Metrics can cause high cpu usage.
> -------------------------------------------------------
>
> Key: HBASE-6602
> URL: https://issues.apache.org/jira/browse/HBASE-6602
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
> Assignee: Elliott Clark
> Priority: Critical
> Fix For: 0.94.2
>
> Attachments: HBASE-6602-0.94-0.patch, HBASE-6602-trunk-0.patch
>
>
> When regions are getting added and removed lots of cpu time can be used by jmx. This is caused by sending jmx messages for every new metric that is added or removed.
> Seeing jstacks like this:
> "RMI TCP Connection(3)-10.4.19.33" daemon prio=10 tid=0x00007f9d64b1d000 nid=0x353 runnable [0x00007f9d598d6000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.getAttribute(MetricsDynamicMBeanBase.java:133)
> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
> at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
> --
> at java.util.TimerThread.run(Timer.java:462)
> "Timer thread for monitoring hbase" daemon prio=10 tid=0x00007f9d648fe000 nid=0x2b5 runnable [0x00007f9d624c7000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.put(HashMap.java:374)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.createMBeanInfo(MetricsDynamicMBeanBase.java:103)
> at org.apache.hadoop.metrics.util.MetricsDynamicMBeanBase.updateMbeanInfoIfMetricsListChanged(MetricsDynamicMBeanBase.java:75)
> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.regionserver.metrics.RegionServerDynamicMetrics.setNumericMetric(RegionServerDynamicMetrics.java:105)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira