You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Nicholas Telford (Jira)" <ji...@apache.org> on 2026/10/06 10:47:00 UTC
[jira] [Updated] (KAFKA-21230) Long rebalances can cause OOM errors with transactional state stores enabled
[ https://issues.apache.org/jira/browse/KAFKA-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicholas Telford updated KAFKA-21230:
-------------------------------------
Description:
When transactional state stores (KIP-892) are enabled, we loosely bound our transaction buffers using the new {{statestore.uncommitted.max.bytes}}, which is checked in {{StreamThread#maybeCommit()}}.
However, {{TaskManager.commit()}} does not guarantee that it commits: if the thread is currently in the middle of a rebalance, *commits are skipped until it completes*. When this happens, processing continues without a commit until the rebalance has completed.
Long rebalances can therefore cause threads to exceed this limit by a large degree, potentially causing OOM errors (native memory, most likely).
I do not think this is a blocker for 4.4.0, because it only affects applications when transactional state stores have been explicitly enabled; and only under very specific circumstances.
was:
When transactional state stores (KIP-892) are enabled, we loosely bound our transaction buffers using the new {{statestore.uncommitted.max.bytes}}, which is checked in {{StreamThread#maybeCommit()}}.
However, {{TaskManager.commit()}} does not guarantee that it commits: if the thread is currently in the middle of a rebalance, *commits are skipped until it completes*. When this happens, processing continues without a commit until the rebalance has completed.
Long rebalances can therefore cause threads to exceed this limit by a large degree, potentially causing OOM errors (native memory, most likely).
> Long rebalances can cause OOM errors with transactional state stores enabled
> ----------------------------------------------------------------------------
>
> Key: KAFKA-21230
> URL: https://issues.apache.org/jira/browse/KAFKA-21230
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.4.0
> Reporter: Nicholas Telford
> Priority: Major
>
> When transactional state stores (KIP-892) are enabled, we loosely bound our transaction buffers using the new {{statestore.uncommitted.max.bytes}}, which is checked in {{StreamThread#maybeCommit()}}.
> However, {{TaskManager.commit()}} does not guarantee that it commits: if the thread is currently in the middle of a rebalance, *commits are skipped until it completes*. When this happens, processing continues without a commit until the rebalance has completed.
> Long rebalances can therefore cause threads to exceed this limit by a large degree, potentially causing OOM errors (native memory, most likely).
> I do not think this is a blocker for 4.4.0, because it only affects applications when transactional state stores have been explicitly enabled; and only under very specific circumstances.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)