You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@kafka.apache.org by "Junyang Liu (Jira)" <ji...@apache.org> on 2022/08/30 12:19:00 UTC

[jira] [Updated] (KAFKA-14189) improve connection limit and reuse of coordinator and leader in KafkaConsumer

     [ https://issues.apache.org/jira/browse/KAFKA-14189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Junyang Liu updated KAFKA-14189:
--------------------------------
    Summary: improve connection limit and reuse of coordinator and leader in KafkaConsumer  (was: connection count limit of coordinator and leader in KafkaConsumer)

> improve connection limit and reuse of coordinator and leader in KafkaConsumer
> -----------------------------------------------------------------------------
>
>                 Key: KAFKA-14189
>                 URL: https://issues.apache.org/jira/browse/KAFKA-14189
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients
>    Affects Versions: 0.9.0.0
>            Reporter: Junyang Liu
>            Priority: Major
>
> The connection id of connection with coordinator in KafkaConsumer is Integer.MAX_VALUE - coordinator id, which is different with connection id of partition leader. So the connection cannot be reused when coordinator and leader are in the same broker, which means we need two seperated connections with the same broker. Suppose such case, a consumer has connected to the coordinator and finished Join and Sync, and wants to send FETCH to leader in the same broker. But the connection count has reached limit, so the consumer with be in the group but cannot consume messages
>  
> The root cause of this issue is that the process of consuming is a combination of multiple connections(connections with coordinator and leader in same broker), not atomic, which may leads to "half connected". I think we can do some improvement:
>  # reuse the connection with coordinator and leader in the same broker
>  # make the connection limit more flexible, such as allowing extra related connections of a consumer when the connection count limit has reached if it has connected to broker



--
This message was sent by Atlassian Jira
(v8.20.10#820010)