[jira] [Created] (KAFKA-12854) Add a config to allow skipping metadata cache update when topic partition is unassigned

["Lincong Li (Jira)" <ji...@apache.org>]

Lincong Li created KAFKA-12854:
----------------------------------

             Summary: Add a config to allow skipping metadata cache update when topic partition is unassigned
                 Key: KAFKA-12854
                 URL: https://issues.apache.org/jira/browse/KAFKA-12854
             Project: Kafka
          Issue Type: Improvement
          Components: clients
            Reporter: Lincong Li


The "assign" method in the consumer triggers a metadata cache update if the new partition assignment is different from the current assignment. It makes sense to update the MD cache if the new assignment contains partitions that do not exist in the current assignment.

However, I wonder why is updating its MD cache necessary if the new partition assignment is a subset of the current assignment. For example, the new assignment is tp0, tp1, and the current assignment is tp0, tp1, tp2.

The current behavior does too many drawbacks in most cases. However, if the number of consumer instances is large and each consumer instance is constantly getting some topic partitions unassigned, the QPS of the MD request sent out to update the MD cache becomes high as a result.
 
Proposed changes:
Add a config to allow skipping metadata cache update when topic partition(s) is unassigned

Existing PR to a forked repo:
https://github.com/linkedin/kafka/pull/166/files



--
This message was sent by Atlassian Jira
(v8.3.4#803005)