You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@kafka.apache.org by "Armand (Jira)" <ji...@apache.org> on 2023/04/04 13:41:00 UTC

[jira] [Created] (KAFKA-14878) Reduce assignment data size to improve Kafka Consumer scalability

Armand created KAFKA-14878:
------------------------------

             Summary: Reduce assignment data size to improve Kafka Consumer scalability
                 Key: KAFKA-14878
                 URL: https://issues.apache.org/jira/browse/KAFKA-14878
             Project: Kafka
          Issue Type: Improvement
          Components: consumer
            Reporter: Armand


This issue is linked to https://issues.apache.org/jira/browse/KAFKA-7149 
which describes scalability problems with Kafka streams and was resolved by this [PR|https://github.com/apache/kafka/pull/7185].

We have the same problem with Kafka Consumer. If we scale our solution with 50 instances (each instance has 5 consumers, so a consumer group can contain 50 * 5 = 250 consumers) with 5000 partitions, the leader of the group will send SyncGroup requests exceeding 100 MB to the group coordinator.

The only solution we found is to increase the max.message.bytes of the __consumer_offsets topic. Is there any plan to compress requests from the consumer in order to avoid sending too big requests over the network?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)