[DISCUSS] Broker behaviour when returning records

["Megan Niu (BLOOMBERG/ TORONTO)" <mn...@bloomberg.net>]

Hi all,
I have some questions about how brokers batch records to send to consumers.

One of the configuration properties for a consumer is fetch.max.bytes.
Here's what "Kafka: The Definitive Guide" (by Shapira et al.) says about fetch.max.bytes :

"This property lets you specify the maximum bytes that Kafka will return whenever the consumer polls a broker (50 MB by default). [...] Note that records are sent to the client in batches, and if the first record-batch that the broker has to send exceeds this size, the batch will be sent and the limit will be ignored."

My questions:
1. Why can't the broker remove messages from the first batch? So that fetch.max.bytes is not violated.
2. How does the broker choose which partitions (from those that the consumer is assigned to) contribute messages to a response batch?
3. Why does the broker send records in batches to the consumer?

Re: [DISCUSS] Broker behaviour when returning records

[Luke Chen <sh...@gmail.com>]

Hi Megan,

Answering your questions below:

1. Why can't the broker remove messages from the first batch? So that
fetch.max.bytes is not violated.

The documentation has answered your question:

If the first record batch in the first non-empty partition of the fetch is
larger than this limit, the batch will still be returned to ensure that the
consumer can make progress.

ref;
https://kafka.apache.org/documentation/#consumerconfigs_max.partition.fetch.bytes


2. How does the broker choose which partitions (from those that the
consumer is assigned to) contribute messages to a response batch?

As far as I know, suppose consumer A subscribed to partition 1,2,3,  the
broker just read as many data as the 1st partition picked (let's say,
partition 1), and not exceed the max fetch limit. If it did not exceed,
continue to the partition 2, and so on.

3. Why does the broker send records in batches to the consumer?

For performance

Thank you.
Luke


On Sat, Jan 22, 2022 at 6:22 AM Megan Niu (BLOOMBERG/ TORONTO) <
mniu25@bloomberg.net> wrote:

> Hi all,
> I have some questions about how brokers batch records to send to consumers.
>
> One of the configuration properties for a consumer is fetch.max.bytes.
> Here's what "Kafka: The Definitive Guide" (by Shapira et al.) says about
> fetch.max.bytes :
>
> "This property lets you specify the maximum bytes that Kafka will return
> whenever the consumer polls a broker (50 MB by default). [...] Note that
> records are sent to the client in batches, and if the first record-batch
> that the broker has to send exceeds this size, the batch will be sent and
> the limit will be ignored."
>
> My questions:
> 1. Why can't the broker remove messages from the first batch? So that
> fetch.max.bytes is not violated.
> 2. How does the broker choose which partitions (from those that the
> consumer is assigned to) contribute messages to a response batch?
> 3. Why does the broker send records in batches to the consumer?