You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Debraj Manna <su...@gmail.com> on 2023/11/30 08:08:01 UTC
Relation between fetch.max.bytes, max.partition.fetch.bytes & max.poll.records
The doc states that fetch.max.bytes & max.partition.fetch.bytes
are not absolute maximum. If the first record batch in the first non-empty
> partition of the fetch is larger than this limit, the batch will still be
> returned to ensure that the consumer can make progress.
I am getting a bit confused.
Let's say I have a configuration like below with sufficient messages in
each partition
- Partitions in a topic 20
- Single message size 2MB
- Consumers 5
- max.poll.records 20
- fetch.max.bytes 50 MB
- max.partition.fetch.bytes 1 MB.
The broker config message.max.bytes and max.message.bytes is set to default
100MB
If the consumer does a poll will it receive 20 records? If yes then there
is no significance of fetch.max.bytes & max.partition.fetch.bytes with
max.poll.records?
- Java Kafka Client - 3.5.1
- Kafka Broker - 2.8.1
Re: Relation between fetch.max.bytes, max.partition.fetch.bytes & max.poll.records
Posted by Haruki Okada <oc...@gmail.com>.
> is there any gain on number of network calls being
made
No basically.
However, since next fetch requests are sent only when previous-fetched
records are processed, setting max.poll.records to too low would negatively
affect the network-call frequency depending on how you process records
Because if every poll() returns only small number of records,
batch-processing (which is known to be efficient) wouldn't be able so
processing previously-fetched records could take time due to the
insufficient throughput.
Also posted on SO: https://stackoverflow.com/a/77633494/3225746
2023年12月10日(日) 3:42 Debraj Manna <su...@gmail.com>:
> Can someone please clarify my below doubt? The same has been asked on stack
> overflow also.
>
> https://stackoverflow.com/q/77630586/785523
>
>
> On Fri, 8 Dec, 2023, 21:33 Debraj Manna, <su...@gmail.com> wrote:
>
> > Thanks again.
> >
> > Another follow-up question, since max.poll.records has nothing to do with
> > fetch requests, then is there any gain on number of network calls being
> > made between consumer & broker if max.poll.records is set to 1 as against
> > let's say the default 500.
> >
> > On Wed, Dec 6, 2023 at 7:21 PM Haruki Okada <oc...@gmail.com> wrote:
> >
> >> poll-idle-ratio-avg=1.0 doesn't immediately mean fetch throughput
> problem
> >> since if processing is very fast, the metric will always be near 1.0.
> >>
> >> 2023年12月4日(月) 13:09 Debraj Manna <su...@gmail.com>:
> >>
> >> > Thanks for the reply.
> >> >
> >> > I read KIP
> >> > <
> >> >
> >>
> https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=127406453#KIP517:Addconsumermetricstoobserveuserpollbehavior-poll-idle-ratio-avg
> >> > >.
> >> > Can you let me know if I observe poll-idle-ration.avg equal to 1.0
> then
> >> > does that mean my fetch.max.bytes or max.partition.fetch.bytes is not
> >> > enough and I have to increase them? If not what could be the reason
> that
> >> > may cause poll-idle-ratio-avg to approach 1.0?
> >> >
> >> >
> >> > Can you let me know what
> >> >
> >> > On Sat, 2 Dec, 2023, 07:05 Haruki Okada, <oc...@gmail.com> wrote:
> >> >
> >> > > Hi.
> >> > >
> >> > > `max.poll.records` does nothing with fetch requests (refs:
> >> > >
> >> > >
> >> >
> >>
> https://kafka.apache.org/35/documentation.html#consumerconfigs_max.poll.records
> >> > > )
> >> > >
> >> > > Then, how many records will be returned for single fetch request
> >> depends
> >> > on
> >> > > the partition-leader assignment. (note: we assume follower-fetch is
> >> not
> >> > > used here)
> >> > > If all partition leaders are in the same broker, 40MB (2MB * 20
> >> > partition)
> >> > > will be returned for a single fetch request.
> >> > >
> >> > > 2023年11月30日(木) 17:10 Debraj Manna <su...@gmail.com>:
> >> > >
> >> > > > The doc states that fetch.max.bytes & max.partition.fetch.bytes
> >> > > >
> >> > > > are not absolute maximum. If the first record batch in the first
> >> > > non-empty
> >> > > > > partition of the fetch is larger than this limit, the batch will
> >> > still
> >> > > be
> >> > > > > returned to ensure that the consumer can make progress.
> >> > > >
> >> > > >
> >> > > > I am getting a bit confused.
> >> > > >
> >> > > > Let's say I have a configuration like below with sufficient
> >> messages in
> >> > > > each partition
> >> > > >
> >> > > >
> >> > > > - Partitions in a topic 20
> >> > > > - Single message size 2MB
> >> > > > - Consumers 5
> >> > > > - max.poll.records 20
> >> > > > - fetch.max.bytes 50 MB
> >> > > > - max.partition.fetch.bytes 1 MB.
> >> > > >
> >> > > > The broker config message.max.bytes and max.message.bytes is set
> to
> >> > > default
> >> > > > 100MB
> >> > > >
> >> > > > If the consumer does a poll will it receive 20 records? If yes
> then
> >> > there
> >> > > > is no significance of fetch.max.bytes & max.partition.fetch.bytes
> >> with
> >> > > > max.poll.records?
> >> > > >
> >> > > >
> >> > > > - Java Kafka Client - 3.5.1
> >> > > > - Kafka Broker - 2.8.1
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > ========================
> >> > > Okada Haruki
> >> > > ocadaruma@gmail.com
> >> > > ========================
> >> > >
> >> >
> >>
> >>
> >> --
> >> ========================
> >> Okada Haruki
> >> ocadaruma@gmail.com
> >> ========================
> >>
> >
>
--
========================
Okada Haruki
ocadaruma@gmail.com
========================
Re: Relation between fetch.max.bytes, max.partition.fetch.bytes & max.poll.records
Posted by Debraj Manna <su...@gmail.com>.
Can someone please clarify my below doubt? The same has been asked on stack
overflow also.
https://stackoverflow.com/q/77630586/785523
On Fri, 8 Dec, 2023, 21:33 Debraj Manna, <su...@gmail.com> wrote:
> Thanks again.
>
> Another follow-up question, since max.poll.records has nothing to do with
> fetch requests, then is there any gain on number of network calls being
> made between consumer & broker if max.poll.records is set to 1 as against
> let's say the default 500.
>
> On Wed, Dec 6, 2023 at 7:21 PM Haruki Okada <oc...@gmail.com> wrote:
>
>> poll-idle-ratio-avg=1.0 doesn't immediately mean fetch throughput problem
>> since if processing is very fast, the metric will always be near 1.0.
>>
>> 2023年12月4日(月) 13:09 Debraj Manna <su...@gmail.com>:
>>
>> > Thanks for the reply.
>> >
>> > I read KIP
>> > <
>> >
>> https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=127406453#KIP517:Addconsumermetricstoobserveuserpollbehavior-poll-idle-ratio-avg
>> > >.
>> > Can you let me know if I observe poll-idle-ration.avg equal to 1.0 then
>> > does that mean my fetch.max.bytes or max.partition.fetch.bytes is not
>> > enough and I have to increase them? If not what could be the reason that
>> > may cause poll-idle-ratio-avg to approach 1.0?
>> >
>> >
>> > Can you let me know what
>> >
>> > On Sat, 2 Dec, 2023, 07:05 Haruki Okada, <oc...@gmail.com> wrote:
>> >
>> > > Hi.
>> > >
>> > > `max.poll.records` does nothing with fetch requests (refs:
>> > >
>> > >
>> >
>> https://kafka.apache.org/35/documentation.html#consumerconfigs_max.poll.records
>> > > )
>> > >
>> > > Then, how many records will be returned for single fetch request
>> depends
>> > on
>> > > the partition-leader assignment. (note: we assume follower-fetch is
>> not
>> > > used here)
>> > > If all partition leaders are in the same broker, 40MB (2MB * 20
>> > partition)
>> > > will be returned for a single fetch request.
>> > >
>> > > 2023年11月30日(木) 17:10 Debraj Manna <su...@gmail.com>:
>> > >
>> > > > The doc states that fetch.max.bytes & max.partition.fetch.bytes
>> > > >
>> > > > are not absolute maximum. If the first record batch in the first
>> > > non-empty
>> > > > > partition of the fetch is larger than this limit, the batch will
>> > still
>> > > be
>> > > > > returned to ensure that the consumer can make progress.
>> > > >
>> > > >
>> > > > I am getting a bit confused.
>> > > >
>> > > > Let's say I have a configuration like below with sufficient
>> messages in
>> > > > each partition
>> > > >
>> > > >
>> > > > - Partitions in a topic 20
>> > > > - Single message size 2MB
>> > > > - Consumers 5
>> > > > - max.poll.records 20
>> > > > - fetch.max.bytes 50 MB
>> > > > - max.partition.fetch.bytes 1 MB.
>> > > >
>> > > > The broker config message.max.bytes and max.message.bytes is set to
>> > > default
>> > > > 100MB
>> > > >
>> > > > If the consumer does a poll will it receive 20 records? If yes then
>> > there
>> > > > is no significance of fetch.max.bytes & max.partition.fetch.bytes
>> with
>> > > > max.poll.records?
>> > > >
>> > > >
>> > > > - Java Kafka Client - 3.5.1
>> > > > - Kafka Broker - 2.8.1
>> > > >
>> > >
>> > >
>> > > --
>> > > ========================
>> > > Okada Haruki
>> > > ocadaruma@gmail.com
>> > > ========================
>> > >
>> >
>>
>>
>> --
>> ========================
>> Okada Haruki
>> ocadaruma@gmail.com
>> ========================
>>
>
Re: Relation between fetch.max.bytes, max.partition.fetch.bytes & max.poll.records
Posted by Debraj Manna <su...@gmail.com>.
Thanks again.
Another follow-up question, since max.poll.records has nothing to do with
fetch requests, then is there any gain on number of network calls being
made between consumer & broker if max.poll.records is set to 1 as against
let's say the default 500.
On Wed, Dec 6, 2023 at 7:21 PM Haruki Okada <oc...@gmail.com> wrote:
> poll-idle-ratio-avg=1.0 doesn't immediately mean fetch throughput problem
> since if processing is very fast, the metric will always be near 1.0.
>
> 2023年12月4日(月) 13:09 Debraj Manna <su...@gmail.com>:
>
> > Thanks for the reply.
> >
> > I read KIP
> > <
> >
> https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=127406453#KIP517:Addconsumermetricstoobserveuserpollbehavior-poll-idle-ratio-avg
> > >.
> > Can you let me know if I observe poll-idle-ration.avg equal to 1.0 then
> > does that mean my fetch.max.bytes or max.partition.fetch.bytes is not
> > enough and I have to increase them? If not what could be the reason that
> > may cause poll-idle-ratio-avg to approach 1.0?
> >
> >
> > Can you let me know what
> >
> > On Sat, 2 Dec, 2023, 07:05 Haruki Okada, <oc...@gmail.com> wrote:
> >
> > > Hi.
> > >
> > > `max.poll.records` does nothing with fetch requests (refs:
> > >
> > >
> >
> https://kafka.apache.org/35/documentation.html#consumerconfigs_max.poll.records
> > > )
> > >
> > > Then, how many records will be returned for single fetch request
> depends
> > on
> > > the partition-leader assignment. (note: we assume follower-fetch is not
> > > used here)
> > > If all partition leaders are in the same broker, 40MB (2MB * 20
> > partition)
> > > will be returned for a single fetch request.
> > >
> > > 2023年11月30日(木) 17:10 Debraj Manna <su...@gmail.com>:
> > >
> > > > The doc states that fetch.max.bytes & max.partition.fetch.bytes
> > > >
> > > > are not absolute maximum. If the first record batch in the first
> > > non-empty
> > > > > partition of the fetch is larger than this limit, the batch will
> > still
> > > be
> > > > > returned to ensure that the consumer can make progress.
> > > >
> > > >
> > > > I am getting a bit confused.
> > > >
> > > > Let's say I have a configuration like below with sufficient messages
> in
> > > > each partition
> > > >
> > > >
> > > > - Partitions in a topic 20
> > > > - Single message size 2MB
> > > > - Consumers 5
> > > > - max.poll.records 20
> > > > - fetch.max.bytes 50 MB
> > > > - max.partition.fetch.bytes 1 MB.
> > > >
> > > > The broker config message.max.bytes and max.message.bytes is set to
> > > default
> > > > 100MB
> > > >
> > > > If the consumer does a poll will it receive 20 records? If yes then
> > there
> > > > is no significance of fetch.max.bytes & max.partition.fetch.bytes
> with
> > > > max.poll.records?
> > > >
> > > >
> > > > - Java Kafka Client - 3.5.1
> > > > - Kafka Broker - 2.8.1
> > > >
> > >
> > >
> > > --
> > > ========================
> > > Okada Haruki
> > > ocadaruma@gmail.com
> > > ========================
> > >
> >
>
>
> --
> ========================
> Okada Haruki
> ocadaruma@gmail.com
> ========================
>
Re: Relation between fetch.max.bytes, max.partition.fetch.bytes & max.poll.records
Posted by Haruki Okada <oc...@gmail.com>.
poll-idle-ratio-avg=1.0 doesn't immediately mean fetch throughput problem
since if processing is very fast, the metric will always be near 1.0.
2023年12月4日(月) 13:09 Debraj Manna <su...@gmail.com>:
> Thanks for the reply.
>
> I read KIP
> <
> https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=127406453#KIP517:Addconsumermetricstoobserveuserpollbehavior-poll-idle-ratio-avg
> >.
> Can you let me know if I observe poll-idle-ration.avg equal to 1.0 then
> does that mean my fetch.max.bytes or max.partition.fetch.bytes is not
> enough and I have to increase them? If not what could be the reason that
> may cause poll-idle-ratio-avg to approach 1.0?
>
>
> Can you let me know what
>
> On Sat, 2 Dec, 2023, 07:05 Haruki Okada, <oc...@gmail.com> wrote:
>
> > Hi.
> >
> > `max.poll.records` does nothing with fetch requests (refs:
> >
> >
> https://kafka.apache.org/35/documentation.html#consumerconfigs_max.poll.records
> > )
> >
> > Then, how many records will be returned for single fetch request depends
> on
> > the partition-leader assignment. (note: we assume follower-fetch is not
> > used here)
> > If all partition leaders are in the same broker, 40MB (2MB * 20
> partition)
> > will be returned for a single fetch request.
> >
> > 2023年11月30日(木) 17:10 Debraj Manna <su...@gmail.com>:
> >
> > > The doc states that fetch.max.bytes & max.partition.fetch.bytes
> > >
> > > are not absolute maximum. If the first record batch in the first
> > non-empty
> > > > partition of the fetch is larger than this limit, the batch will
> still
> > be
> > > > returned to ensure that the consumer can make progress.
> > >
> > >
> > > I am getting a bit confused.
> > >
> > > Let's say I have a configuration like below with sufficient messages in
> > > each partition
> > >
> > >
> > > - Partitions in a topic 20
> > > - Single message size 2MB
> > > - Consumers 5
> > > - max.poll.records 20
> > > - fetch.max.bytes 50 MB
> > > - max.partition.fetch.bytes 1 MB.
> > >
> > > The broker config message.max.bytes and max.message.bytes is set to
> > default
> > > 100MB
> > >
> > > If the consumer does a poll will it receive 20 records? If yes then
> there
> > > is no significance of fetch.max.bytes & max.partition.fetch.bytes with
> > > max.poll.records?
> > >
> > >
> > > - Java Kafka Client - 3.5.1
> > > - Kafka Broker - 2.8.1
> > >
> >
> >
> > --
> > ========================
> > Okada Haruki
> > ocadaruma@gmail.com
> > ========================
> >
>
--
========================
Okada Haruki
ocadaruma@gmail.com
========================
Re: Relation between fetch.max.bytes, max.partition.fetch.bytes & max.poll.records
Posted by Debraj Manna <su...@gmail.com>.
Thanks for the reply.
I read KIP
<https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=127406453#KIP517:Addconsumermetricstoobserveuserpollbehavior-poll-idle-ratio-avg>.
Can you let me know if I observe poll-idle-ration.avg equal to 1.0 then
does that mean my fetch.max.bytes or max.partition.fetch.bytes is not
enough and I have to increase them? If not what could be the reason that
may cause poll-idle-ratio-avg to approach 1.0?
Can you let me know what
On Sat, 2 Dec, 2023, 07:05 Haruki Okada, <oc...@gmail.com> wrote:
> Hi.
>
> `max.poll.records` does nothing with fetch requests (refs:
>
> https://kafka.apache.org/35/documentation.html#consumerconfigs_max.poll.records
> )
>
> Then, how many records will be returned for single fetch request depends on
> the partition-leader assignment. (note: we assume follower-fetch is not
> used here)
> If all partition leaders are in the same broker, 40MB (2MB * 20 partition)
> will be returned for a single fetch request.
>
> 2023年11月30日(木) 17:10 Debraj Manna <su...@gmail.com>:
>
> > The doc states that fetch.max.bytes & max.partition.fetch.bytes
> >
> > are not absolute maximum. If the first record batch in the first
> non-empty
> > > partition of the fetch is larger than this limit, the batch will still
> be
> > > returned to ensure that the consumer can make progress.
> >
> >
> > I am getting a bit confused.
> >
> > Let's say I have a configuration like below with sufficient messages in
> > each partition
> >
> >
> > - Partitions in a topic 20
> > - Single message size 2MB
> > - Consumers 5
> > - max.poll.records 20
> > - fetch.max.bytes 50 MB
> > - max.partition.fetch.bytes 1 MB.
> >
> > The broker config message.max.bytes and max.message.bytes is set to
> default
> > 100MB
> >
> > If the consumer does a poll will it receive 20 records? If yes then there
> > is no significance of fetch.max.bytes & max.partition.fetch.bytes with
> > max.poll.records?
> >
> >
> > - Java Kafka Client - 3.5.1
> > - Kafka Broker - 2.8.1
> >
>
>
> --
> ========================
> Okada Haruki
> ocadaruma@gmail.com
> ========================
>
Re: Relation between fetch.max.bytes, max.partition.fetch.bytes & max.poll.records
Posted by Haruki Okada <oc...@gmail.com>.
Hi.
`max.poll.records` does nothing with fetch requests (refs:
https://kafka.apache.org/35/documentation.html#consumerconfigs_max.poll.records
)
Then, how many records will be returned for single fetch request depends on
the partition-leader assignment. (note: we assume follower-fetch is not
used here)
If all partition leaders are in the same broker, 40MB (2MB * 20 partition)
will be returned for a single fetch request.
2023年11月30日(木) 17:10 Debraj Manna <su...@gmail.com>:
> The doc states that fetch.max.bytes & max.partition.fetch.bytes
>
> are not absolute maximum. If the first record batch in the first non-empty
> > partition of the fetch is larger than this limit, the batch will still be
> > returned to ensure that the consumer can make progress.
>
>
> I am getting a bit confused.
>
> Let's say I have a configuration like below with sufficient messages in
> each partition
>
>
> - Partitions in a topic 20
> - Single message size 2MB
> - Consumers 5
> - max.poll.records 20
> - fetch.max.bytes 50 MB
> - max.partition.fetch.bytes 1 MB.
>
> The broker config message.max.bytes and max.message.bytes is set to default
> 100MB
>
> If the consumer does a poll will it receive 20 records? If yes then there
> is no significance of fetch.max.bytes & max.partition.fetch.bytes with
> max.poll.records?
>
>
> - Java Kafka Client - 3.5.1
> - Kafka Broker - 2.8.1
>
--
========================
Okada Haruki
ocadaruma@gmail.com
========================