[jira] [Comment Edited] (KAFKA-656) Add Quotas to Kafka

["Abhinav Anand (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14100679#comment-14100679 ] 

Abhinav Anand edited comment on KAFKA-656 at 8/18/14 2:22 PM:
--------------------------------------------------------------

Hi [~jkreps], 

 We also want to avoid using zookeeper as dependency. With the number of producers being pretty high, I am planning to follow the approach that has been proposed in this thread. I can create a design to get my approach reviewed. I am creating a QuotaManager thread on each broker. This process would be responsible to verify the quota permissions for each client at partition-topic level. Any thing that I should be aware of to avoid affecting producer performance ? Throwing the QuotaExceeded exception and will catch it on the producer itself. 
Let me know any suggestions that you have, I will try to incorporate in the design. 

Regards,
Abhinav


was (Author: ab10anand):
Hi [~jkreps], 

 We also want to avoid using zookeeper as dependency. With the number of producers being pretty high, I am planning to follow the approach that has been proposed in this thread. I can create a design to get my approach reviewed. I am creating a QuotaManager thread on each broker. This process would be responsible to verify the quota permissions for each client at partition-topic level. Any thing that I should be aware of to avoid affecting producer performance. Throwing the QuotaExceeded exception and will catch it on the producer itself. 
Let me know any suggestions that you have, I will try to incorporate in the design. 

Regards,
Abhinav

> Add Quotas to Kafka
> -------------------
>
>                 Key: KAFKA-656
>                 URL: https://issues.apache.org/jira/browse/KAFKA-656
>             Project: Kafka
>          Issue Type: New Feature
>          Components: core
>    Affects Versions: 0.8.1
>            Reporter: Jay Kreps
>              Labels: project
>
> It would be nice to implement a quota system in Kafka to improve our support for highly multi-tenant usage. The goal of this system would be to prevent one naughty user from accidently overloading the whole cluster.
> There are several quantities we would want to track:
> 1. Requests pers second
> 2. Bytes written per second
> 3. Bytes read per second
> There are two reasonable groupings we would want to aggregate and enforce these thresholds at:
> 1. Topic level
> 2. Client level (e.g. by client id from the request)
> When a request hits one of these limits we will simply reject it with a QUOTA_EXCEEDED exception.
> To avoid suddenly breaking things without warning, we should ideally support two thresholds: a soft threshold at which we produce some kind of warning and a hard threshold at which we give the error. The soft threshold could just be defined as 80% (or whatever) of the hard threshold.
> There are nuances to getting this right. If you measure second-by-second a single burst may exceed the threshold, so we need a sustained measurement over a period of time.
> Likewise when do we stop giving this error? To make this work right we likely need to charge against the quota for request *attempts* not just successful requests. Otherwise a client that is overloading the server will just flap on and off--i.e. we would disable them for a period of time but when we re-enabled them they would likely still be abusing us.
> It would be good to a wiki design on how this would all work as a starting point for discussion.



--
This message was sent by Atlassian JIRA
(v6.2#6252)