[jira] [Created] (HBASE-12097) Blocked threads on hbase, slowly increasing, appears to be updating metrics

["Michael Smith (JIRA)" <ji...@apache.org>]

Michael Smith created HBASE-12097:
-------------------------------------

             Summary: Blocked threads on hbase, slowly increasing, appears to be updating metrics
                 Key: HBASE-12097
                 URL: https://issues.apache.org/jira/browse/HBASE-12097
             Project: HBase
          Issue Type: Bug
          Components: regionserver
    Affects Versions: 0.94.6
         Environment: RHEL 6.2, CDH 4.3.0.0
            Reporter: Michael Smith


Hbase shows an increasing number of IPC Threads in BLOCKED state

Hundreds of these,more and more appearing over hours, performance degrading, requiring regionserver restart to restore performance.

Thread:

Thread 421 (IPC Server handler 368 on 60201):
  State: BLOCKED
  Blocked count: 19314
  Waited count: 322565
  Blocked on org.apache.hadoop.metrics.util.MetricsIntValue@1ec5ca55
  Blocked by 236 (IPC Server handler 183 on 60201)
  Stack:
    org.apache.hadoop.metrics.util.MetricsIntValue.set(MetricsIntValue.java:73)
org.apache.hadoop.hbase.ipc.HBaseServer.updateCallQueueLenMetrics(HBaseServer.java:1360)   org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1399)

i dont actually know how to troubleshoot this much further... Happy to take suggestions...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)