You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Swapnil Ghike (JIRA)" <ji...@apache.org> on 2013/07/13 23:35:48 UTC
[jira] [Commented] (KAFKA-656) Add Quotas to Kafka
[ https://issues.apache.org/jira/browse/KAFKA-656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707836#comment-13707836 ]
Swapnil Ghike commented on KAFKA-656:
-------------------------------------
Hey Prashanth/Jonathan, have you had the time to run with this? If you're busy, I would be interested in taking this up.
> Add Quotas to Kafka
> -------------------
>
> Key: KAFKA-656
> URL: https://issues.apache.org/jira/browse/KAFKA-656
> Project: Kafka
> Issue Type: New Feature
> Components: core
> Affects Versions: 0.8.1
> Reporter: Jay Kreps
> Labels: project
>
> It would be nice to implement a quota system in Kafka to improve our support for highly multi-tenant usage. The goal of this system would be to prevent one naughty user from accidently overloading the whole cluster.
> There are several quantities we would want to track:
> 1. Requests pers second
> 2. Bytes written per second
> 3. Bytes read per second
> There are two reasonable groupings we would want to aggregate and enforce these thresholds at:
> 1. Topic level
> 2. Client level (e.g. by client id from the request)
> When a request hits one of these limits we will simply reject it with a QUOTA_EXCEEDED exception.
> To avoid suddenly breaking things without warning, we should ideally support two thresholds: a soft threshold at which we produce some kind of warning and a hard threshold at which we give the error. The soft threshold could just be defined as 80% (or whatever) of the hard threshold.
> There are nuances to getting this right. If you measure second-by-second a single burst may exceed the threshold, so we need a sustained measurement over a period of time.
> Likewise when do we stop giving this error? To make this work right we likely need to charge against the quota for request *attempts* not just successful requests. Otherwise a client that is overloading the server will just flap on and off--i.e. we would disable them for a period of time but when we re-enabled them they would likely still be abusing us.
> It would be good to a wiki design on how this would all work as a starting point for discussion.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (KAFKA-656) Add Quotas to Kafka
Posted by S Ahmed <sa...@gmail.com>.
This thread might be of some use, it discusses the memory footprint when
using metrics:
https://groups.google.com/forum/#!topic/metrics-user/T68H70ea1Og
On Sat, Jul 13, 2013 at 5:35 PM, Swapnil Ghike (JIRA) <ji...@apache.org>wrote:
>
> [
> https://issues.apache.org/jira/browse/KAFKA-656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707836#comment-13707836]
>
> Swapnil Ghike commented on KAFKA-656:
> -------------------------------------
>
> Hey Prashanth/Jonathan, have you had the time to run with this? If you're
> busy, I would be interested in taking this up.
>
> > Add Quotas to Kafka
> > -------------------
> >
> > Key: KAFKA-656
> > URL: https://issues.apache.org/jira/browse/KAFKA-656
> > Project: Kafka
> > Issue Type: New Feature
> > Components: core
> > Affects Versions: 0.8.1
> > Reporter: Jay Kreps
> > Labels: project
> >
> > It would be nice to implement a quota system in Kafka to improve our
> support for highly multi-tenant usage. The goal of this system would be to
> prevent one naughty user from accidently overloading the whole cluster.
> > There are several quantities we would want to track:
> > 1. Requests pers second
> > 2. Bytes written per second
> > 3. Bytes read per second
> > There are two reasonable groupings we would want to aggregate and
> enforce these thresholds at:
> > 1. Topic level
> > 2. Client level (e.g. by client id from the request)
> > When a request hits one of these limits we will simply reject it with a
> QUOTA_EXCEEDED exception.
> > To avoid suddenly breaking things without warning, we should ideally
> support two thresholds: a soft threshold at which we produce some kind of
> warning and a hard threshold at which we give the error. The soft threshold
> could just be defined as 80% (or whatever) of the hard threshold.
> > There are nuances to getting this right. If you measure second-by-second
> a single burst may exceed the threshold, so we need a sustained measurement
> over a period of time.
> > Likewise when do we stop giving this error? To make this work right we
> likely need to charge against the quota for request *attempts* not just
> successful requests. Otherwise a client that is overloading the server will
> just flap on and off--i.e. we would disable them for a period of time but
> when we re-enabled them they would likely still be abusing us.
> > It would be good to a wiki design on how this would all work as a
> starting point for discussion.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>