You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "nicktelford (via GitHub)" <gi...@apache.org> on 2026/04/24 16:31:36 UTC
[PR] KAFKA-20489: Add enable.transactional.statestores [kafka]
nicktelford opened a new pull request, #22141:
URL: https://github.com/apache/kafka/pull/22141
Introduces the `enable.transactional.statestores` config (default false). When enabled, uncommitted writes are held in an in-memory buffer per store and are not flushed to the underlying base store until the Kafka transaction commits, making staged writes invisible to IQ reads at the committed isolation level until the containing commit completes.
The config is threaded through TopologyConfig, TaskConfig, task creators, and ProcessorStateManager. When transactional stores are active, the EOS state wipe on unclean shutdown is suppressed: since uncommitted data never reaches the base store, there is nothing to wipe; corruption is handled explicitly via markChangelogAsCorrupted.
StreamTask.postCommit is extended to flush the pending write buffer on every commit interval, not only on task revocation or close. Under EOS the normal commit-interval path previously skipped maybeCheckpoint, the only path that calls stateMgr.commit() to flush each store's buffer. Without this, READ_COMMITTED IQ readers see no new data mid-run, and the uncommitted buffer grows unbounded between task lifecycle events.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20489: Add enable.transactional.statestores [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on PR #22141:
URL: https://github.com/apache/kafka/pull/22141#issuecomment-4359724499
Kicked off a system test - will post the results here later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20489: Add enable.transactional.statestores [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on code in PR #22141:
URL: https://github.com/apache/kafka/pull/22141#discussion_r3170631836
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -1107,6 +1115,11 @@ public class StreamsConfig extends AbstractConfig {
true,
Importance.LOW,
ENABLE_METRICS_PUSH_DOC)
+ .define(TRANSACTIONAL_STATE_STORES_CONFIG,
Review Comment:
I think we want some sort of check in `StreamsConfig` that will throw if users set `TRANSACTIONAL_STATE_STORES_CONFIG` with out setting `PROCESSING_GUARANTEE` to `EXACTLY_ONCE_V2`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20489: Add enable.transactional.statestores [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on PR #22141:
URL: https://github.com/apache/kafka/pull/22141#issuecomment-4409706698
[System tests for this PR passed](https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2026-05-08--001.88af2385-7e68-4133-b506-43744a246719--1778269493--nicktelford--KIP-892-txn-config--fe1b52d347/report.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20489: Add enable.transactional.statestores [kafka]
Posted by "nicktelford (via GitHub)" <gi...@apache.org>.
nicktelford commented on code in PR #22141:
URL: https://github.com/apache/kafka/pull/22141#discussion_r3189574910
##########
streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java:
##########
@@ -1107,6 +1115,11 @@ public class StreamsConfig extends AbstractConfig {
true,
Importance.LOW,
ENABLE_METRICS_PUSH_DOC)
+ .define(TRANSACTIONAL_STATE_STORES_CONFIG,
Review Comment:
You can enable it even under ALOS. Doing so should yield improved throughput for RocksDB stores, because writes are buffered in a `WriteBatch`, instead of writing to the underlying RocksDB database directly, which incurs locking overhead on every write.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20489: Add enable.transactional.statestores [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck merged PR #22141:
URL: https://github.com/apache/kafka/pull/22141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org