[jira] [Updated] (KAFKA-19462) "fetch.max.bytes" config is not honored when remote + local fetch

["Luke Chen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Luke Chen updated KAFKA-19462:
------------------------------
    Description: 
Currently in local fetch case, we'll calculate the remaining bytes to be fetched for each partition via "fetch.max.bytes" and "max.partition.fetch.bytes" configs. For example:
 # Config:
max.partition.fetch.bytes = 1MB
fetch.max.bytes = 1.5MB
 # Topic foo has 2 partitions.
 # Consumer fetches data from topic foo
 # Fetches from foo-0 first, it got 1MB of data (max.partition.fetch.bytes), so remaining 0.5 MB of data available to be fetched
 # Fetches from foo-1 for max 0.5MB.
 # Total returned 1.5MB records

However, in remote + local fetch case, because we don't know how much data we can fetch before querying remote log metadata manager or other resource, we can't have a value to tell replicaManager beforehand. Currently, we treat it as 0 bytes read. And that's why the final returned data could exceed the "fetch.max.bytes" value.

For example:
 # Config:
max.partition.fetch.bytes = 1MB
fetch.max.bytes = 1.5MB
 # Topic foo has 2 partitions + topic boo has 1 partition with tiered storage enabled.
 # Consumer fetches data from topic foo and boo
 # Fetches from boo-0, because we don't know how much data we can get, return 0, and send to remote async read.
 # Fetches from foo-0, it got 1MB of data, so remaining 0.5 MB of data available to be fetched
 # Fetches from foo-1 for max 0.5MB.
 # remote async read for boo-0, and it got 1MB data (max.partition.fetch.bytes).
 # Total returned 2.5MB records, which exceeds `fetch.max.bytes = 1.5MB`

 

  was:
Currently in local fetch case, we'll calculate the remaining bytes to be fetched for each partition via "fetch.max.bytes" and "max.partition.fetch.bytes" configs. For example:
 # Config: 
max.partition.fetch.bytes = 1MB
fetch.max.bytes = 1.5MB
 # Topic foo has 2 partitions.
 # Consumer fetches data from topic foo
 # Fetches from foo-0 first, it got 1MB of data, so remaining 0.5 MB of data available to be fetched
 # Fetches from foo-1 for max 0.5MB.
 # Total returned 1.5MB records

However, in remote + local fetch case, because we don't know how much data we can fetch before querying remote log metadata manager or other resource, we can't have a value to tell replicaManager beforehand. Currently, we treat it as 0 bytes read. And that's why the final returned data could exceed the "fetch.max.bytes" value. 

For example:
 # Config: 
max.partition.fetch.bytes = 1MB
fetch.max.bytes = 1.5MB
 # Topic foo has 2 partitions + topic boo has 1 partition with tiered storage enabled.
 # Consumer fetches data from topic foo and boo
 # Fetches from boo-0, because we don't know how much data we can get, return 0, and send to remote async read.
 # Fetches from foo-0, it got 1MB of data, so remaining 0.5 MB of data available to be fetched
 # Fetches from foo-1 for max 0.5MB.
 # remote async read for boo-0, and it got 1MB data (max.partition.fetch.bytes).
 # Total returned 2.5MB records, which exceeds `fetch.max.bytes = 1.5MB`

 


> "fetch.max.bytes" config is not honored when remote + local fetch
> -----------------------------------------------------------------
>
>                 Key: KAFKA-19462
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19462
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Luke Chen
>            Assignee: Luke Chen
>            Priority: Major
>
> Currently in local fetch case, we'll calculate the remaining bytes to be fetched for each partition via "fetch.max.bytes" and "max.partition.fetch.bytes" configs. For example:
>  # Config:
> max.partition.fetch.bytes = 1MB
> fetch.max.bytes = 1.5MB
>  # Topic foo has 2 partitions.
>  # Consumer fetches data from topic foo
>  # Fetches from foo-0 first, it got 1MB of data (max.partition.fetch.bytes), so remaining 0.5 MB of data available to be fetched
>  # Fetches from foo-1 for max 0.5MB.
>  # Total returned 1.5MB records
> However, in remote + local fetch case, because we don't know how much data we can fetch before querying remote log metadata manager or other resource, we can't have a value to tell replicaManager beforehand. Currently, we treat it as 0 bytes read. And that's why the final returned data could exceed the "fetch.max.bytes" value.
> For example:
>  # Config:
> max.partition.fetch.bytes = 1MB
> fetch.max.bytes = 1.5MB
>  # Topic foo has 2 partitions + topic boo has 1 partition with tiered storage enabled.
>  # Consumer fetches data from topic foo and boo
>  # Fetches from boo-0, because we don't know how much data we can get, return 0, and send to remote async read.
>  # Fetches from foo-0, it got 1MB of data, so remaining 0.5 MB of data available to be fetched
>  # Fetches from foo-1 for max 0.5MB.
>  # remote async read for boo-0, and it got 1MB data (max.partition.fetch.bytes).
>  # Total returned 2.5MB records, which exceeds `fetch.max.bytes = 1.5MB`
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)