[jira] [Commented] (ZOOKEEPER-1383) Create update throughput quotas and add hard quota limits

["Thawan Kooburat (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581871#comment-13581871 ] 

Thawan Kooburat commented on ZOOKEEPER-1383:
--------------------------------------------

RFC: 

Existing quota logic in ZooKeeper only used for keeping track of node count and byte usage per path.  When “soft” limit is exceeded, the server log warning message.  This is not sufficient for our operation requirement.  Here is want we are planning to do. We already implemented majority of these functionalities except the hard limit.  

1. Resource metric – The system will be able to monitor the following resource usage and enforce hard limit. 
- node count 
- used bytes
- write throughput (bytes/sec, transactions/sec)
- read throughput (bytes/sec, transactions/sec)  (monitoring only, no hard limit)

2. Usage monitoring and soft-limit
The server is going to export per-path usage statistic via four-letter command.  Since this is easier for external monitoring system to get these numbers than reading from ZooKeeper directly.  For example, the new command can report the following stats

used_bytes.<path-A>     300
used_bytes.<path-B>     500
node_count.<path-A>     20
node_count.<path-B>     40
read_bytes.<path-A>.60  20     //Read byte/sec for the last 1 min
read_bytes.<path-B>.60  20

For read throughput and write throughput, all servers will report read throughput statistics but only the leader report write throughput statistic.  Internally, we already used a high performance multi windows counters provided by Facebook’s jcommon (https://github.com/facebook/jcommon/blob/master/stats/src/main/java/com/facebook/stats/MultiWindowRate.java) However, I think the community may want a simpler counter to reduce the dependency requirement. 

Additional, I am going to add an option to disable soft-limit check since writing warning message to log file is not that useful and may affect performance (especially when replying txnlog and soft limit is exceeded).  

3.  Hard limit
PrepRequestProcessor on the leader have to decide when to reject a given request (instead of the current patch that rejects the request down in DataTree).  However, PrepRequestProcessor will need to access more data from the DataTree in order to decide when to reject a request. There are 2 possible implementations here.  First, usage tracking and limit checking is implemented in the DataTree, this is simpler given the amount of information that is available in DataTree itself. The problem is that this limit checking logic will not be accurate when there are in-flight requests. The other option is to move limit checking to PrepRequestProcessor and maintain in-flight statistic similar to ChangeRecord. 

I think the later option is much more complicate. Since it is ok to allow usage to exceed hard limit slightly, I might go with the first option. 

Let me know if you have any suggestion. 

                
> Create update throughput quotas and add hard quota limits
> ---------------------------------------------------------
>
>                 Key: ZOOKEEPER-1383
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1383
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: server
>            Reporter: Jay Shrauner
>            Assignee: Thawan Kooburat
>         Attachments: ZOOKEEPER-1383.patch, ZOOKEEPER-1383.patch, ZOOKEEPER-1383.patch, ZOOKEEPER-1383.patch
>
>
> Quotas exist for size (node count and size in bytes); it would be useful to track and support quotas on update throughput (bytes per second) as well. This can be tracked on both a node/subtree level for quota support as well as on the server level for monitoring.
> In addition, the existing quotas log a warning when they are exceeded but allow the transaction to proceed (soft quotas). It would also be useful to support a corresponding set of hard quota limits that fail the transaction.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira