[jira] [Resolved] (HBASE-12097) Blocked threads on hbase, slowly increasing, appears to be updating metrics

["Andrew Purtell (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-12097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell resolved HBASE-12097.
------------------------------------
    Resolution: Invalid

Please write to user@hbase.apache.org for troubleshooting assistance.

> Blocked threads on hbase, slowly increasing, appears to be updating metrics
> ---------------------------------------------------------------------------
>
>                 Key: HBASE-12097
>                 URL: https://issues.apache.org/jira/browse/HBASE-12097
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.94.6
>         Environment: RHEL 6.2, CDH 4.3.0.0
>            Reporter: Michael Smith
>         Attachments: total_blocks_versus_threads-week.png
>
>
> Hbase shows an increasing number of IPC Threads in BLOCKED state
> Hundreds of these,more and more appearing over hours, performance degrading, requiring regionserver restart to restore performance.
> Thread:
> Thread 421 (IPC Server handler 368 on 60201):
>   State: BLOCKED
>   Blocked count: 19314
>   Waited count: 322565
>   Blocked on org.apache.hadoop.metrics.util.MetricsIntValue@1ec5ca55
>   Blocked by 236 (IPC Server handler 183 on 60201)
>   Stack:
>     org.apache.hadoop.metrics.util.MetricsIntValue.set(MetricsIntValue.java:73)
> org.apache.hadoop.hbase.ipc.HBaseServer.updateCallQueueLenMetrics(HBaseServer.java:1360)   org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1399)
> i dont actually know how to troubleshoot this much further... Happy to take suggestions...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)