Requires suggestions for Producer request throttling

["sukumar.np" <su...@zohocorp.com>]

Hi Team,



We are having a Kafka cluster with multiple Topics in it and shared with multiple services(clients). Each service will have multiple events source from where they will be pushing messages to Kafka brokers. Once a service starts producing message at a high rate, it will affect other services(clients) because it will fill the disk quickly which will bring the cluster down. So, we want to throttle Producer request once it crosses the specified threshold(Which can be set on Topic basis, not for each service).



After checking with Quota feature available in Kafka we found that It allows pushing data to a queue and will keep responses in delay queue(if it requires getting throttled). If we apply quota for our use case then below problems can happen:

a).  Since quota observes message rate for a window and starts to throttle the producer responses, meanwhile all incoming messages will be added to the queue. It may fill the disks quickly as there are many producers for the same Topics and will create an outage for Kafka service.

b). For sync Producers, because of throttling response will be delayed, which will result in hang the user-thread or app-servers.



So we don't want to go for applying quota for our use case. Can you please share some suggestions to handle this use-case in our Kafka broker. Like, before messages get appended to log, it should validate for throttling and if it requires being throttled. Throttling mechanism should be either slow down the request rate up to specified time frame or throw some generic exception from broker side to clients.



Our Kafka setup like,

Having 3 brokers in a cluster and each Topic has replication factor 3 and using Kafka-0.10.0.1.



Looking forward to your suggestions.



Thanks

Sukumar N

Re: Requires suggestions for Producer request throttling

[suku march <su...@gmail.com>]

Hi hans,

Thanks for your response. We are already looked this KIP-124 - Request rate quotas in Kafka 0.11.0, But it will help us to throttle based on request processing time taken by the user/client or broker CPU utilization of request thread. 

We are looking for some mechanism, like which will throttle the producer request only, since it causes the service outage due to heavy load of messages. So we have to throttle the producer request based on a volume of messages before pushing into a queue. We are probably okay with some less amount of latency for this implementation.

We are looking for your suggestions.

Thanks
Sukumar N

On 2017-06-29 20:34 (+0530), Hans Jespersen <ha...@confluent.io> wrote: 
> Request quotas was just added to 0.11. Does that help in your use case?
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+Request+rate+quotas
> 
> -hans
> 
> > On Jun 29, 2017, at 12:55 AM, sukumar.np <su...@zohocorp.com> wrote:
> > 
> > Hi Team,
> > 
> > 
> > 
> > We are having a Kafka cluster with multiple Topics in it and shared with multiple services(clients). Each service will have multiple events source from where they will be pushing messages to Kafka brokers. Once a service starts producing message at a high rate, it will affect other services(clients) because it will fill the disk quickly which will bring the cluster down. So, we want to throttle Producer request once it crosses the specified threshold(Which can be set on Topic basis, not for each service).
> > 
> > 
> > 
> > After checking with Quota feature available in Kafka we found that It allows pushing data to a queue and will keep responses in delay queue(if it requires getting throttled). If we apply quota for our use case then below problems can happen:
> > 
> > a).  Since quota observes message rate for a window and starts to throttle the producer responses, meanwhile all incoming messages will be added to the queue. It may fill the disks quickly as there are many producers for the same Topics and will create an outage for Kafka service.
> > 
> > b). For sync Producers, because of throttling response will be delayed, which will result in hang the user-thread or app-servers.
> > 
> > 
> > 
> > So we don't want to go for applying quota for our use case. Can you please share some suggestions to handle this use-case in our Kafka broker. Like, before messages get appended to log, it should validate for throttling and if it requires being throttled. Throttling mechanism should be either slow down the request rate up to specified time frame or throw some generic exception from broker side to clients.
> > 
> > 
> > 
> > Our Kafka setup like,
> > 
> > Having 3 brokers in a cluster and each Topic has replication factor 3 and using Kafka-0.10.0.1.
> > 
> > 
> > 
> > Looking forward to your suggestions.
> > 
> > 
> > 
> > Thanks
> > 
> > Sukumar N
> > 
> > 
> > 
> > 
> > 
> 

Re: Requires suggestions for Producer request throttling

[Hans Jespersen <ha...@confluent.io>]

Request quotas was just added to 0.11. Does that help in your use case?

https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+Request+rate+quotas

-hans

> On Jun 29, 2017, at 12:55 AM, sukumar.np <su...@zohocorp.com> wrote:
> 
> Hi Team,
> 
> 
> 
> We are having a Kafka cluster with multiple Topics in it and shared with multiple services(clients). Each service will have multiple events source from where they will be pushing messages to Kafka brokers. Once a service starts producing message at a high rate, it will affect other services(clients) because it will fill the disk quickly which will bring the cluster down. So, we want to throttle Producer request once it crosses the specified threshold(Which can be set on Topic basis, not for each service).
> 
> 
> 
> After checking with Quota feature available in Kafka we found that It allows pushing data to a queue and will keep responses in delay queue(if it requires getting throttled). If we apply quota for our use case then below problems can happen:
> 
> a).  Since quota observes message rate for a window and starts to throttle the producer responses, meanwhile all incoming messages will be added to the queue. It may fill the disks quickly as there are many producers for the same Topics and will create an outage for Kafka service.
> 
> b). For sync Producers, because of throttling response will be delayed, which will result in hang the user-thread or app-servers.
> 
> 
> 
> So we don't want to go for applying quota for our use case. Can you please share some suggestions to handle this use-case in our Kafka broker. Like, before messages get appended to log, it should validate for throttling and if it requires being throttled. Throttling mechanism should be either slow down the request rate up to specified time frame or throw some generic exception from broker side to clients.
> 
> 
> 
> Our Kafka setup like,
> 
> Having 3 brokers in a cluster and each Topic has replication factor 3 and using Kafka-0.10.0.1.
> 
> 
> 
> Looking forward to your suggestions.
> 
> 
> 
> Thanks
> 
> Sukumar N
> 
> 
> 
> 
>