[jira] [Updated] (ZOOKEEPER-759) Stop accepting connections when close to file descriptor limit

["Michael Han (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Han updated ZOOKEEPER-759:
----------------------------------
    Fix Version/s:     (was: 3.5.3)
                   3.5.4

> Stop accepting connections when close to file descriptor limit
> --------------------------------------------------------------
>
>                 Key: ZOOKEEPER-759
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-759
>             Project: ZooKeeper
>          Issue Type: Improvement
>          Components: server
>            Reporter: Travis Crawford
>             Fix For: 3.5.4, 3.6.0
>
>
> Zookeeper always tries to accept new connections, throwing an exception if out of file descriptors. An improvement would be denying new client connections when close to the limit.
> Additionally, file-descriptor limits+usage should be exported to the monitoring four-letter word, should that get implemented (see ZOOKEEPER-744).
> DETAILS
> A Zookeeper ensemble I administer recently suffered an outage when one node was restarted with the low system-default ulimit of 1024 file descriptors and later ran out. File descriptor usage+max are already being monitored by the following MBeans:
> - java.lang.OperatingSystem.MaxFileDescriptorCount
> - java.lang.OperatingSystem.OpenFileDescriptorCount
> They're described (rather tersely) at:
> http://java.sun.com/javase/6/docs/jre/api/management/extension/com/sun/management/UnixOperatingSystemMXBean.html
> This feature request is for the following:
> (a) Stop accepting new connections when OpenFileDescriptorCount is close to MaxFileDescriptorCount, defaulting to 95% FD usage. New connections should be denied, logged to disk at debug level, and increment a ``ConnectionDeniedCount`` MBean counter.
> (b) Begin accepting new connections when usage drops below some configurable threshold, defaulting to 90% of FD usage, basically the high/low watermark model.
> (c) Update the administrators guide with a comment about using an appropriate FD limit.
> (d) Extra credit: if ZOOKEEPER-744 is implemented export statistics for:
> zookeeper_open_file_descriptor_count
> zookeeper_max_file_descriptor_count
> zookeeper_max_file_descriptor_mismatch - boolean, exported by leader, if not all zk's have the same max FD value



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)