You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "bbejeck (via GitHub)" <gi...@apache.org> on 2026/04/27 20:35:09 UTC
[PR] MINOR: Add docs for KIP-1035 [kafka]
bbejeck opened a new pull request, #22161:
URL: https://github.com/apache/kafka/pull/22161
Add documentation for the changes brought by KIP-1035
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add docs for KIP-1035 [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on PR #22161:
URL: https://github.com/apache/kafka/pull/22161#issuecomment-4352615347
@mjsax ready for another review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add docs for KIP-1035 [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on PR #22161:
URL: https://github.com/apache/kafka/pull/22161#issuecomment-4361065013
Merged #22161 into trunk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add docs for KIP-1035 [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on PR #22161:
URL: https://github.com/apache/kafka/pull/22161#issuecomment-4361124416
Cherry-picked to 4.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add docs for KIP-1035 [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #22161:
URL: https://github.com/apache/kafka/pull/22161#discussion_r3157330981
##########
docs/streams/upgrade-guide.md:
##########
@@ -71,6 +78,8 @@ The streams thread metrics `commit-ratio`, `process-ratio`, `punctuate-ratio`, a
Kafka Streams now allows to purge local state directories and checkpoint files during application startup if they have not been modified for a certain period of time. This can be configured via the new `state.cleanup.dir.max.age.ms` config. More details can be found in [KIP-1259](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1259%3A+Add+configuration+to+wipe+Kafka+Streams+local+state+on+startup)
+Kafka Streams now persists state store changelog offsets inside each state store rather than in a single per-task `.checkpoint` file, as described in [KIP-1035](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1035%3A+StateStore+managed+changelog+offsets). For built-in (RocksDB) stores, offsets are written into a dedicated column family alongside store data. Persistent stores that do not manage their own offsets (including custom user-implemented stores) are transparently wrapped to maintain a per-store `.checkpoint_<storeName>` file, preserving the existing behavior. Existing per-task `.checkpoint` files are migrated automatically on first startup and then deleted; no application changes or operator action is required to upgrade. This change is largely internal infrastructure — and importantly, **it does not yet change behavior under the `exactly-once` `processing.guarantee`**: on a crash or processing error, state stores are still wiped and fully restored from the changelog
topics, the same as in 4.2 and earlier. KIP-1035 is a precursor and prerequisite for [KIP-892: Transactional Semantics for StateStores](https://cwiki.apache.org/confluence/display/KAFKA/KIP-892%3A+Transactional+Semantics+for+StateStores), which will leverage these per-store offsets to make state store writes transactional with the changelog and avoid full restoration after EOS errors. Until KIP-892 lands, the user-visible impact of KIP-1035 is limited to the on-disk layout change and its downgrade implications — see [Notable compatibility changes in past releases](#notable-compatibility-changes-in-past-releases) for the procedure to downgrade to 4.2.x or earlier. Custom `StateStore` implementations may opt-in to managing their own offsets by overriding `managesOffsets()` to return `true`, implementing `commit(Map<TopicPartition, Long>)` to atomically persist offsets with their data, and implementing `committedOffset(TopicPartition)` to return the most-recently-committed offset fo
r a partition; otherwise the wrapper continues to handle checkpointing on their behalf.
Review Comment:
I agree with Nick that his paragraph is a little bit lengthy. I would only say on a high-level that checkpoint file moves into the store. Maybe keep a short note about KIP-892, but only brief. And say, "internal change", not relevant for most users; only relevant if you implement a custom state-store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add docs for KIP-1035 [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck merged PR #22161:
URL: https://github.com/apache/kafka/pull/22161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add docs for KIP-1035 [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #22161:
URL: https://github.com/apache/kafka/pull/22161#discussion_r3157318316
##########
docs/streams/upgrade-guide.md:
##########
@@ -57,6 +57,13 @@ Downgrading from 3.0.x or newer version to 2.8.x or older version needs special
Downgrading from 4.0.x or newer version to older than 4.0.0 needs special attention: Since 4.0.0 release, Kafka Streams upgraded RocksDB from version 7.9.2 to 9.7.3. This upgrade introduces a RocksDB file format version change from version 5 to version 6 (introduced in RocksDB 8.6). While the newer RocksDB version (9.7.3) can read state stores written in the old format (version 5), the older RocksDB versions cannot read state stores written in the new format (version 6). This means that downgrading from Kafka Streams 4.0.x or newer to versions older than 4.0.0 in-flight is not possible out-of-the-box. Users must first delete the local RocksDB state stores written by Kafka Streams 4.0.x or newer before downgrading to versions older than 4.0.0, which will then restore the state stores from the changelogs using the old file format.
+Downgrading from 4.3.x or newer version to 4.2.x or older version needs special attention: Since 4.3.0 release, Kafka Streams persists state store changelog offsets inside each state store rather than in a per-task `.checkpoint` file (see [KIP-1035](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1035%3A+StateStore+managed+changelog+offsets)). On startup, the legacy per-task `.checkpoint` file is migrated into the per-store offset metadata and then deleted, so older Kafka Streams versions that still expect the legacy file will not find their offsets after a downgrade and will treat the state stores as uninitialized — forcing a full restore from the changelog topics. To support an in-flight downgrade without restoring state, 4.3.0+ will write the legacy consolidated per-task `.checkpoint` file back out on a clean shutdown when `upgrade.from` is set to a version less than 4.3. Perform the downgrade as two rolling bounces:
+
+ * In the first rolling bounce, on each application instance running 4.3.0+, set `upgrade.from="4.2"` (or earlier, e.g. `"4.1"`, `"4.0"`) and restart the instance. Each instance must be cleanly shut down at least once with this config set so that an up-to-date legacy `.checkpoint` file is written into every task and global state directory alongside the per-store offsets.
+ * In the second rolling bounce, swap the binaries to the older version (4.2.x or earlier) and remove the `upgrade.from` config. The older version will read the legacy `.checkpoint` file as it always did and resume from the persisted offsets.
+
+If an instance is downgraded without first being cleanly shut down with `upgrade.from` set (for example, after a crash, or by skipping the first rolling bounce), the legacy `.checkpoint` file will be missing or stale, and the older version will restore the affected stores in full from the changelog topics. See [KAFKA-19710](https://issues.apache.org/jira/browse/KAFKA-19710) for the change that introduced this downgrade path.
Review Comment:
The missing checkpoint file is only have of the issue, right? When we do a clean downgrade, we also delete the metadata CF inside RocksDB? Hence, if downgraded "brute force" RocksDB can't even be opened and KS would crash, right, and it would be required to cleanup the state directory (what implies even if a checkpoint file would be there, it would get deleted).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add docs for KIP-1035 [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on code in PR #22161:
URL: https://github.com/apache/kafka/pull/22161#discussion_r3161946123
##########
docs/streams/upgrade-guide.md:
##########
@@ -57,6 +57,13 @@ Downgrading from 3.0.x or newer version to 2.8.x or older version needs special
Downgrading from 4.0.x or newer version to older than 4.0.0 needs special attention: Since 4.0.0 release, Kafka Streams upgraded RocksDB from version 7.9.2 to 9.7.3. This upgrade introduces a RocksDB file format version change from version 5 to version 6 (introduced in RocksDB 8.6). While the newer RocksDB version (9.7.3) can read state stores written in the old format (version 5), the older RocksDB versions cannot read state stores written in the new format (version 6). This means that downgrading from Kafka Streams 4.0.x or newer to versions older than 4.0.0 in-flight is not possible out-of-the-box. Users must first delete the local RocksDB state stores written by Kafka Streams 4.0.x or newer before downgrading to versions older than 4.0.0, which will then restore the state stores from the changelogs using the old file format.
+Downgrading from 4.3.x or newer version to 4.2.x or older version needs special attention: Since 4.3.0 release, Kafka Streams persists state store changelog offsets inside each state store rather than in a per-task `.checkpoint` file (see [KIP-1035](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1035%3A+StateStore+managed+changelog+offsets)). On startup, the legacy per-task `.checkpoint` file is migrated into the per-store offset metadata and then deleted, so older Kafka Streams versions that still expect the legacy file will not find their offsets after a downgrade and will treat the state stores as uninitialized — forcing a full restore from the changelog topics. To support an in-flight downgrade without restoring state, 4.3.0+ will write the legacy consolidated per-task `.checkpoint` file back out on a clean shutdown when `upgrade.from` is set to a version less than 4.3. Perform the downgrade as two rolling bounces:
+
+ * In the first rolling bounce, on each application instance running 4.3.0+, set `upgrade.from="4.2"` (or earlier, e.g. `"4.1"`, `"4.0"`) and restart the instance. Each instance must be cleanly shut down at least once with this config set so that an up-to-date legacy `.checkpoint` file is written into every task and global state directory alongside the per-store offsets.
+ * In the second rolling bounce, swap the binaries to the older version (4.2.x or earlier) and remove the `upgrade.from` config. The older version will read the legacy `.checkpoint` file as it always did and resume from the persisted offsets.
+
+If an instance is downgraded without first being cleanly shut down with `upgrade.from` set (for example, after a crash, or by skipping the first rolling bounce), the legacy `.checkpoint` file will be missing or stale, and the older version will restore the affected stores in full from the changelog topics. See [KAFKA-19710](https://issues.apache.org/jira/browse/KAFKA-19710) for the change that introduced this downgrade path.
Review Comment:
Yes, you are correct in the downgrade path we don't drop the offsets column family on close. I've updated the text to reflect users will need to wipe the state directories when downgrading and I'll file a Jira ticket to add the drop column family support on downgrading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org