[jira] [Commented] (KAFKA-4405) Kafka consumer improperly send prefetch request

["Jason Gustafson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15700704#comment-15700704 ] 

Jason Gustafson commented on KAFKA-4405:
----------------------------------------

I think a better approach is KAFKA-4007. The idea is to always send the next prefetch (regardless whether we still have data queued), but avoid reading it from the socket until we need to. This is also related to KIP-81: https://cwiki.apache.org/confluence/display/KAFKA/KIP-81%3A+Bound+Fetch+memory+usage+in+the+consumer.

> Kafka consumer improperly send prefetch request
> -----------------------------------------------
>
>                 Key: KAFKA-4405
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4405
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.10.0.1
>            Reporter: ysysberserk
>
> Now kafka consumer has added max.poll.records to limit the count of messages return by poll().
> According to KIP-41, to implement  max.poll.records, the prefetch request should only be sent when the total number of retained records is less than max.poll.records.
> But in the code of 0.10.0.1 , the consumer will send a prefetch request if it retained any records and never check if total number of retained records is less than max.poll.records..
> If max.poll.records is set to a count much less than the count of message fetched , the poll() loop will send a lot of requests than expected and will have more and more records fetched and stored in memory before they can be consumed.
> So before sending a  prefetch request , the consumer must check if total number of retained records is less than max.poll.records.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)