[jira] [Created] (KAFKA-9677) Low consume bandwidth quota may cause consumer not being able to fetch data

["Anna Povzner (Jira)" <ji...@apache.org>]

Anna Povzner created KAFKA-9677:
-----------------------------------

             Summary: Low consume bandwidth quota may cause consumer not being able to fetch data
                 Key: KAFKA-9677
                 URL: https://issues.apache.org/jira/browse/KAFKA-9677
             Project: Kafka
          Issue Type: Bug
          Components: core
    Affects Versions: 2.3.1, 2.4.0, 2.2.2, 2.1.1, 2.0.1
            Reporter: Anna Povzner
            Assignee: Anna Povzner


When we changed quota communication with KIP-219, fetch requests get throttled by returning empty response with the delay in `throttle_time_ms` and Kafka consumer retrying again after the delay. 

With default configs, the maximum fetch size could be as big as 50MB (or 10MB per partition). The default broker config (1-second window, 10 full windows of tracked bandwidth/thread utilization usage) means that < 5MB/s consumer quota (per broker) may stop fetch request from ever being successful.

Or the other way around: 1 MB/s consumer quota (per broker) means that any fetch request that gets >= 10MB of data (10 seconds * 1MB/second) in the response will never get through.
h3. Proposed fix

Return less data in fetch response in this case: Cap `fetchMaxBytes` passed to replicaManager.fetchMessages() from KafkaApis.handleFetchRequest() to <tracking window> * <consume bandwidth quota>. In the example of default configs and 1MB/s consumer bandwidth quota, fetchMaxBytes will be 10MB.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)