You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Steven Schlansker (Jira)" <ji...@apache.org> on 2025/09/05 19:32:00 UTC

[jira] [Created] (KAFKA-19678) Streams open iterator tracking has high contention on metrics lock

Steven Schlansker created KAFKA-19678:
-----------------------------------------

             Summary: Streams open iterator tracking has high contention on metrics lock
                 Key: KAFKA-19678
                 URL: https://issues.apache.org/jira/browse/KAFKA-19678
             Project: Kafka
          Issue Type: Bug
          Components: streams
    Affects Versions: 4.1.0
            Reporter: Steven Schlansker
         Attachments: image-2025-09-05-12-13-24-910.png

We run Kafka Streams 4.1.0 with custom processors that heavily use state store range iterators.

While investigating disappointing performance, we found a surprising source of lock contention.

Over the course of about a 1 minute profiler sample, the {{org.apache.kafka.common.metrics.Metrics}} lock is taken approximately 40,000 times and blocks threads for about 1 minute.

This appears to be because our state stores generally have no iterators open, except when their processor is processing a record, in which case it opens an iterator (taking the lock through {{OpenIterators.add}} into {{{}Metrics.registerMetric{}}}), does a tiny bit of work, and then closes the iterator (again taking the lock through {{OpenIterators.remove}} into {{{}Metrics.removeMetric{}}}).

So, stream processing threads takes a globally shared lock twice per record, for this subset of our data. I've attached a profiler thread state visualization with our findings - the red bar indicates the thread was blocked during the sample on this lock. As you can see, this lock seems to be severely hampering our performance.

 

!image-2025-09-05-12-13-24-910.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)