[jira] [Updated] (KAFKA-7639) Read one request at a time from socket to reduce broker memory usage

["Matthias J. Sax (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-7639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias J. Sax updated KAFKA-7639:
-----------------------------------
    Fix Version/s:     (was: 2.2.0)

> Read one request at a time from socket to reduce broker memory usage
> --------------------------------------------------------------------
>
>                 Key: KAFKA-7639
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7639
>             Project: Kafka
>          Issue Type: Improvement
>          Components: network
>            Reporter: Rajini Sivaram
>            Assignee: Rajini Sivaram
>            Priority: Major
>
> Broker's Selector currently reads all requests available on the socket when the socket is ready for read. These are queued up as staged receives. We mute the channel and stop reading any more data until all the staged requests are processed. This behaviour is slightly inconsistent since for the initial read we drain the socket buffer, allowing it to get filled up again, but if data arrives slighly after the initial read, then we dont read from the socket buffer until pending requests are processed.
> To avoid holding onto requests for longer than required, we should read one request at a time even if more data is available in the socket buffer. This is especially useful for produce requests which may be large and may take long to process.
> Note that with the default socket read buffer size of 100K, this is not a critical issue. But with larger socket buffers, this could result in excessive memory usage if a lot of produce requests are buffered in the broker and the producer times out, reconnects and sends more data before broker has cleared older requests. By reading one-at-a-time, we reduce the amount of time the broker holds onto memory for each request.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)