[jira] [Created] (KAFKA-1016) Broker should limit purgatory size

["Chris Riccomini (JIRA)" <ji...@apache.org>]

Chris Riccomini created KAFKA-1016:
--------------------------------------

             Summary: Broker should limit purgatory size
                 Key: KAFKA-1016
                 URL: https://issues.apache.org/jira/browse/KAFKA-1016
             Project: Kafka
          Issue Type: Bug
          Components: purgatory
    Affects Versions: 0.8
            Reporter: Chris Riccomini
            Assignee: Joel Koshy


I recently ran into a case where a poorly configured Kafka consumer was able to trigger out of memory exceptions in multiple Kafka brokers. The consumer was configured to have a fetcher.max.wait of Int.MaxInt.

For low volume topics, this configuration causes the consumer to block for frequently, and for long periods of time. [~junrao] informs me that the fetch request will time out after the socket timeout is reached. In our case, this was set to 30s.

With several thousand consumer threads, the fetch request purgatory got into the 100,000-400,000 range, which we believe triggered the out of memory exception. [~nehanarkhede] claims to have seem similar behavior in other high volume clusters.

It kind of seems like a bad thing that a poorly configured consumer can trigger out of memory exceptions in the broker. I was thinking maybe it makes sense to have the broker try and protect itself from this situation. Here are some potential solutions:

1. Have a broker-side max wait config for fetch requests.
2. Threshold the purgatory size, and either drop the oldest connections in purgatory, or reject the newest fetch requests when purgatory is full.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira