[jira] [Resolved] (KAFKA-19460) fetch result might have size < fetch.min.bytes even if data is available in replica

["Luke Chen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-19460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Chen resolved KAFKA-19460.
-------------------------------
    Fix Version/s: 4.2.0
       Resolution: Fixed

> fetch result might have size < fetch.min.bytes even if data is available in replica 
> ------------------------------------------------------------------------------------
>
>                 Key: KAFKA-19460
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19460
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Luke Chen
>            Priority: Major
>             Fix For: 4.2.0
>
>
> In the doc of "[fetch.min.bytes|https://kafka.apache.org/documentation/#consumerconfigs_fetch.min.bytes]", it said:
> ??The minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait for that much data to accumulate before answering the request.??
> It makes users believe the records returned will always greater fetch.min.bytes if there is sufficient data in replica. But even if the data is sufficient is available in the replica, there is still possible the returned records size < fetch.min.bytes.
>  
> For example:
>  # Config 
> fetch.max.bytes=1500
> max.partition.fetch.bytes=1000
> fetch.min.bytes=1100
> fetch.max.wait.ms=500
>  # topic foo has 2 partitions, and each partition contains 1 record with size 1000 bytes.
>  # When a consumer fetches data from these 2 partitions, it starts from foo-0, and fetch 1000 bytes of data, and 500 bytes left before reaching fetch.max.bytes.
>  # When fetching foo-1, since we only have 500 bytes available to be fetched, and the first batch size in foo-1 is 1000 bytes, which is greater than 500, so we don't fetch it.
>  # In the end, the total returned size is 1000 bytes, which is less than fetch.min.bytes, without waiting until `fetch.max.wait.ms` expired. It's because we checked the total size in replicas are more than "fetch.min.bytes", so no wait for "fetch.max.wait.ms".
>  
> I think the logic is correct. It's just we need to update the doc to make it clear to users. We might also need to check `replica.fetch.min.bytes` config.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)