[jira] Commented: (ZOOKEEPER-883) Idle cluster increasingly consumes CPU resources

["Vishal K (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965374#action_12965374 ] 

Vishal K commented on ZOOKEEPER-883:
------------------------------------

Hi,

If this problem is reproducible, can you try with the patch attached for ZOOKEEPER-880. Going through the logs, the patch for ZOOKEEPER-880 will only fix part of the problem - it will prevent leak of SendWorker thread.

Logs show that there are 320 RecvWorker threads blocked in a read() from remote socket. 2 of these threads should be legitimate threads to remote servers. Very likely rest of them are from Nagios.

There are 2 cases I can think of that can lead to this situation:
1. Nagios may not be closing a connection. If Nagios was closing connections, then the read() should have received an exception.
2. Monitoring frequency set too high

-Vishal


> Idle cluster increasingly consumes CPU resources
> ------------------------------------------------
>
>                 Key: ZOOKEEPER-883
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-883
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: server
>    Affects Versions: 3.3.1
>            Reporter: Lars George
>         Attachments: Archive.zip
>
>
> Monitoring the ZooKeeper nodes by polling the various ports using Nagios' open port checks seems to cause a substantial raise of CPU being used by the ZooKeeper daemons. Over the course of a week an idle cluster grew from a baseline 2% to >10% CPU usage. Attached is a stack dump and logs showing the occupied threads. At the end the daemon starts failing on "too many open files" errors as all handles are used up.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.