Logstash 1.5.3 with Kafka 0.9’s quota

[Debraj Manna <su...@gmail.com>]

In Kafka 0.9 support for quota is added as mentioned here
<http://kafka.apache.org/090/documentation.html#design_quotas>. I am having
some doubts about the behavior of quota in accordance with logstash.

So in Kafka 0.9 quota can be configured for each client-id. If one of the
client violates the quota then the kafka broker does not return an error
rather it attempts to slow down a client exceeding its quota. It computes
the amount of delay needed to bring a guilty client under it's quota and
delays the response for that time.

Now the doubt I am having is let's say a single instance of logstash is
pushing logs from two different services (logs are pushed via
logstash-forwarder) to kafka with clientIds A & B respectively. Client A is
sending data under its quota while Client B is exceeding its quota.
Since client
B violates the quota as configured in Kafka then Kafka will delay the
response to slow down B to bring its usage under quota. This will in turn
exert a back-pressure on the logstash-forwarder & slow its log forwarding
to logstash.

Due to this slow down

   1. Will the logstash-forwarder, who is forwarding logs for Client A (who
   is not violating the quota ) be effected in anyway or
   2. The slow down of logstash will effect both Client A & Client B or
   3. This will slow down all logstash outputs not just restricted to Kafka
   Outputs