You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "sstremler (via GitHub)" <gi...@apache.org> on 2026/04/06 19:25:07 UTC
[PR] KAFKA-20398: Fix memory leak by removing StreamsThreadMetricsDelegatingReporter from metrics registry on thread shutdown [kafka]
sstremler opened a new pull request, #21973:
URL: https://github.com/apache/kafka/pull/21973
[KAFKA-20398](https://issues.apache.org/jira/browse/KAFKA-20398)
### Problem
When a stream thread in Kafka Streams crashes, `StreamsUncaughtExceptionHandler` can be configured to replace the crashed thread with a new one (`StreamThreadExceptionResponse.REPLACE_THREAD`). Each time a new stream thread is created, it receives the same shared metrics registry and adds a new `StreamsThreadMetricsDelegatingReporter` to it. However, when the old thread shuts down, its reporter is never removed from the registry. Over time, the registry accumulates one reporter per thread replacement, each holding a reference to its thread's KafkaConsumer, eventually causing an `OutOfMemoryError`.
### Solution
Store the `StreamsThreadMetricsDelegatingReporter` instance as a field on `StreamThread`, populated in the `create()` method after the reporter is registered. During `completeShutdown()`, remove the reporter from the metrics registry before the thread transitions to `DEAD`.
### Verification
I added a test to verify that exactly one reporter exists after thread creation and zero reporters remain after shutdown.
I ran the reproducer application for one hour with continuous thread replacements. Heap analysis confirmed that only two `StreamsThreadMetricsDelegatingReporter` instances were present in the metrics registry — one for the active `StreamThread` and one for the `GlobalStreamThread` — regardless of how many thread replacements had occurred. Before the fix, the count grew unboundedly with each replacement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20398: Fix memory leak by removing StreamsThreadMetricsDelegatingReporter from metrics registry on thread shutdown [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on PR #21973:
URL: https://github.com/apache/kafka/pull/21973#issuecomment-4201498125
Merged #21971 into trunk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20398: Fix memory leak by removing StreamsThreadMetricsDelegatingReporter from metrics registry on thread shutdown [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on PR #21973:
URL: https://github.com/apache/kafka/pull/21973#issuecomment-4202413434
@sstremler I tried cherry-picking to 4.2 but there are some conflcts, can you do a separate PR for 4.2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20398: Fix memory leak by removing StreamsThreadMetricsDelegatingReporter from metrics registry on thread shutdown [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on PR #21973:
URL: https://github.com/apache/kafka/pull/21973#issuecomment-4202405099
cherry-picked to 4.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20398: Fix memory leak by removing StreamsThreadMetricsDelegatingReporter from metrics registry on thread shutdown [kafka]
Posted by "sstremler (via GitHub)" <gi...@apache.org>.
sstremler commented on PR #21973:
URL: https://github.com/apache/kafka/pull/21973#issuecomment-4202809570
> @sstremler I tried cherry-picking to 4.2 but there are some conflcts, can you do a separate PR for 4.2?
@bbejeck Sure: https://github.com/apache/kafka/pull/21989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20398: Fix memory leak by removing StreamsThreadMetricsDelegatingReporter from metrics registry on thread shutdown [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck merged PR #21973:
URL: https://github.com/apache/kafka/pull/21973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org