[jira] [Comment Edited] (KAFKA-12431) Fetch Request/Response without Topic information

["Tamas Kornai (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-12431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298866#comment-17298866 ] 

Tamas Kornai edited comment on KAFKA-12431 at 3/10/21, 2:26 PM:
----------------------------------------------------------------

The issue turned out to be related to how we defined default quotas. We used the following settings:

{{kafka-configs --zookeeper <zk-address>:2181 --alter --add-config 'producer_byte_rate=0, consumer_byte_rate=0, request_percentage=1000' --entity-type users --entity-default}}

In version 2.4 the `0` rate was interpreted as enabling the tracking of quota metrics but no actual limits were enforced. From >=2.5 versions the `0` limit does trigger rate limiting on clients. Setting more reasonable (i.e. large) producer/consumer rate values fixed the high CPU issue that we observed.


was (Author: tkornai):
The issue turned out to be related to how we defined default quotas. We used the following settings:

{{kafka-configs --zookeeper <zk-address>:2181 --alter --add-config 'producer_byte_rate=0, consumer_byte_rate=0, request_percentage=1000' --entity-type users --entity-default}}

In version 2.4 the `0` rate was interpreted as enabling the tracking of quota metrics but no actual limits were enforced. From >=2.5 versions the `0` limit does trigger rate limiting on consumers. Setting more reasonable (i.e. large) producer/consumer rate values fixed the high CPU issue that we observed.

> Fetch Request/Response without Topic information
> ------------------------------------------------
>
>                 Key: KAFKA-12431
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12431
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 2.6.1
>            Reporter: Peter Sinoros-Szabo
>            Priority: Major
>         Attachments: fetch-on-2.4.1.png, fetch-on-2.6.1.png, kafka-highcpu-24.svg.zip, kafka-highcpu-26.svg.zip
>
>
> I was running a 6 node Kafka 2.4.1 cluster with protocol and message format version set to 2.4. I wanted to upgrade the cluster to 2.6.1 and after I upgraded the 1st broker to 2.6.1 without any configuration change, I noticed much higher CPU usage on that broker (instead of 25% CPU usage it was  ~350%) and about 3-4x higher network traffic. So I dumped the traffic between the Kafka client and broker and compared it with the traffic of the same broker downgraded to 2.4.1.
> It seems to me that after I upgraded to 2.6.1, the Fetch requests and responses are not complete, it is missing the topics part of the Fetch Request, I don't know for what reason. I guess there should be always a topics part.
> I'll attache a screenshot from these messages.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)