You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "zhaobo (JIRA)" <ji...@apache.org> on 2019/06/16 06:03:00 UTC

[jira] [Created] (KAFKA-8543) Optimize Fetch thread allocation strategy to ensure full utilization of Fetch thread resources

zhaobo created KAFKA-8543:
-----------------------------

             Summary: Optimize Fetch thread allocation strategy to ensure full utilization of Fetch thread resources
                 Key: KAFKA-8543
                 URL: https://issues.apache.org/jira/browse/KAFKA-8543
             Project: Kafka
          Issue Type: Improvement
          Components: core
    Affects Versions: 2.2.0
            Reporter: zhaobo


The Follow TP on the Broker needs to fetch data from the Leader broker. The original fetch thread allocation strategy has two areas that can be optimized:
(1)The configuration file has a parameter "num.consumer.fetchers" which limits the number of threads fetching data from a source broker. In fact, Fetch resources are not fully used, and only a part of the threads will be used with a high probability. When the traffic of the Leader TP increases, the fetch capability may be bottlenecked.
(2)The number of TPs responsible for each Fetch thread may be uneven.
Improve: We propose a polling allocation method to ensure that Fetch thread resources can be fully utilized. And the number of TPs responsible for each fetch thread is balanced.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)