Question about REQUEST QUEUE size

[Huân La Quốc Nhựt <hu...@hcmut.edu.vn.INVALID>]

Hi guys,
According to the document we can get the request queue size (the one
network threads enqueue, IO threads dequeue and will block network thread
if full) via "kafka.network:type=RequestChannel,name=RequestQueueSize".

However, when I use jconsole to view this metric it always = 0. I have
configured *"num.io.threads=1"* to decrease the dequeue operation as much
as possible and generated ~ 85k records/s from producers (payloads are just
few bytes < 100 bytes for each record) but this metric still remains the
same.

Is there anything wrong with this metric and its exporter or kafka io
threads are just basically too fast that I can not catch up the change in
its size. If this metric is broken, are there any alternatives should I
consider to get Kafka current request queue size ?

I have tested in Kafka 3.3 and 3.4 (the latest one).
I am looking forward to receiving any replies ASAP. Thanks all.

Re: Question about REQUEST QUEUE size

[Huân La Quốc Nhựt <hu...@hcmut.edu.vn.INVALID>]

Thanks for your reply, after that I figured out my total producer rate is
just around 8000/s although message rate is around 80K - 100K (message in
per sec metric). Is 8000 requests per sec are simply too tiny to have any
significant changes in request queue size ?

May I ask you some additional questions?
- What is the difference between message and request cause I sent ~ 80 -
100K records per sec with java API with streaming *linger.ms
<http://linger.ms>=0* (get this number from producer metrics *records send
rate*) but it only converts to 8000 requests per sec ? Some terminologies
in Kafka are quite confusing to me.
- The metric
"kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent"
value is from 0 to 1 (based on documentation) but why it ≈ 2 in my case.

Thanks again for your help.

Vào Th 3, 7 thg 3, 2023 vào lúc 21:39 Divij Vaidya <
divijvaidya13@gmail.com> đã viết:

> The metric that you are monitoring is correct and works correctly in my
> setup for the versions that you mentioned.
>
> 1. To rule out any problems with reading via jconsole, do you see a
> non-zero value for
>
> "kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=([-.\w]+)"
> ?
> 2. Do you observe that all handler threads are busy? You can verify this by
> looking at
>
> "kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent"
> and ensuring it's around 0.
> 3. Are your produce requests getting throttled or dropped due to
> authentication/authorization errors? You can verify this by looking at
>
> "kafka.network:type=RequestMetrics,name=ErrorsPerSec,request=([-.\w]+),error=([-.\w]+)"
>
> --
> Divij Vaidya
>
>
>
> On Tue, Mar 7, 2023 at 11:43 AM Huân La Quốc Nhựt
> <hu...@hcmut.edu.vn.invalid> wrote:
>
> > Hi guys,
> > According to the document we can get the request queue size (the one
> > network threads enqueue, IO threads dequeue and will block network thread
> > if full) via "kafka.network:type=RequestChannel,name=RequestQueueSize".
> >
> > However, when I use jconsole to view this metric it always = 0. I have
> > configured *"num.io.threads=1"* to decrease the dequeue operation as much
> > as possible and generated ~ 85k records/s from producers (payloads are
> just
> > few bytes < 100 bytes for each record) but this metric still remains the
> > same.
> >
> > Is there anything wrong with this metric and its exporter or kafka io
> > threads are just basically too fast that I can not catch up the change in
> > its size. If this metric is broken, are there any alternatives should I
> > consider to get Kafka current request queue size ?
> >
> > I have tested in Kafka 3.3 and 3.4 (the latest one).
> > I am looking forward to receiving any replies ASAP. Thanks all.
> >
>

Re: Question about REQUEST QUEUE size

[Divij Vaidya <di...@gmail.com>]

The metric that you are monitoring is correct and works correctly in my
setup for the versions that you mentioned.

1. To rule out any problems with reading via jconsole, do you see a
non-zero value for
"kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec,topic=([-.\w]+)"
?
2. Do you observe that all handler threads are busy? You can verify this by
looking at
"kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent"
and ensuring it's around 0.
3. Are your produce requests getting throttled or dropped due to
authentication/authorization errors? You can verify this by looking at
"kafka.network:type=RequestMetrics,name=ErrorsPerSec,request=([-.\w]+),error=([-.\w]+)"

--
Divij Vaidya



On Tue, Mar 7, 2023 at 11:43 AM Huân La Quốc Nhựt
<hu...@hcmut.edu.vn.invalid> wrote:

> Hi guys,
> According to the document we can get the request queue size (the one
> network threads enqueue, IO threads dequeue and will block network thread
> if full) via "kafka.network:type=RequestChannel,name=RequestQueueSize".
>
> However, when I use jconsole to view this metric it always = 0. I have
> configured *"num.io.threads=1"* to decrease the dequeue operation as much
> as possible and generated ~ 85k records/s from producers (payloads are just
> few bytes < 100 bytes for each record) but this metric still remains the
> same.
>
> Is there anything wrong with this metric and its exporter or kafka io
> threads are just basically too fast that I can not catch up the change in
> its size. If this metric is broken, are there any alternatives should I
> consider to get Kafka current request queue size ?
>
> I have tested in Kafka 3.3 and 3.4 (the latest one).
> I am looking forward to receiving any replies ASAP. Thanks all.
>