[jira] [Comment Edited] (KAFKA-19678) Streams open iterator tracking has high contention on metrics lock

["Matthias J. Sax (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-19678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18029618#comment-18029618 ] 

Matthias J. Sax edited comment on KAFKA-19678 at 10/13/25 10:37 PM:
--------------------------------------------------------------------

We did not observe any regression in our test/benchmark setup with regard to throughput (but maybe our setup is just not catching it), and nobody reported anything about it yet (well, beside you :))... But yes, might be worth to look into. It's not just FK join, but also session-windows, sliding-windows, and stream-stream join that use range scans... (maybe also others – would need to double check the code).


was (Author: mjsax):
We did not observe any regression in our test/benchmark setup with regard to throughput, and nobody reported anything about it yet... But yes, might be worth to look into. It's not just FK join, but also session-windows, sliding-windows, and stream-stream join that use range scans... (maybe also others – would need to double check the code).

> Streams open iterator tracking has high contention on metrics lock
> ------------------------------------------------------------------
>
>                 Key: KAFKA-19678
>                 URL: https://issues.apache.org/jira/browse/KAFKA-19678
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 4.1.0
>            Reporter: Steven Schlansker
>            Priority: Major
>         Attachments: image-2025-09-05-12-13-24-910.png
>
>
> We run Kafka Streams 4.1.0 with custom processors that heavily use state store range iterators.
> While investigating disappointing performance, we found a surprising source of lock contention.
> Over the course of about a 1 minute profiler sample, the {{org.apache.kafka.common.metrics.Metrics}} lock is taken approximately 40,000 times and blocks threads for about 1 minute.
> This appears to be because our state stores generally have no iterators open, except when their processor is processing a record, in which case it opens an iterator (taking the lock through {{OpenIterators.add}} into {{{}Metrics.registerMetric{}}}), does a tiny bit of work, and then closes the iterator (again taking the lock through {{OpenIterators.remove}} into {{{}Metrics.removeMetric{}}}).
> So, stream processing threads takes a globally shared lock twice per record, for this subset of our data. I've attached a profiler thread state visualization with our findings - the red bar indicates the thread was blocked during the sample on this lock. As you can see, this lock seems to be severely hampering our performance.
>  
> !image-2025-09-05-12-13-24-910.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)