[jira] [Resolved] (KAFKA-20804) Reduce lock contention in ProducerMetadata#add

["Kamal Chandraprakash (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-20804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kamal Chandraprakash resolved KAFKA-20804.
------------------------------------------
    Fix Version/s: 4.4.0
       Resolution: Fixed

> Reduce lock contention in ProducerMetadata#add
> ----------------------------------------------
>
>                 Key: KAFKA-20804
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20804
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients
>            Reporter: Xi Wang
>            Priority: Critical
>             Fix For: 4.4.0
>
>
> add() is called on every send() via KafkaProducer.waitOnMetadata, and the synchronized method acquiring the instance lock causing synchronized contention which impacts the performance.
> Switch the topics map to a ConcurrentHashMap and refresh known topics via a lock-free replace(), which only writes if the topic is still present. For a topic seen for the first time (or one concurrently evicted by retainTopic()), fall back to a synchronized block so the map insert and newTopics bookkeeping happen atomically with retainTopic() - otherwise retainTopic() could expire-and-remove the topic in the gap between the two, leaving it recorded in newTopics without the corresponding map entry, or silently re-inserting an evicted topic without the newTopics bookkeeping needed to trigger an immediate refresh.
> retainTopic() switches to a conditional remove(topic, expireMs) so a concurrent refresh of an existing topic in add() can't be undone by an eviction decision based on the expiry value read just before the refresh landed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)