You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "vcrfxia (via GitHub)" <gi...@apache.org> on 2023/03/24 01:04:51 UTC
[GitHub] [kafka] vcrfxia opened a new pull request, #13444: KAFKA-14491: [21/N] Docs updates for versioned state stores
vcrfxia opened a new pull request, #13444:
URL: https://github.com/apache/kafka/pull/13444
(This PR is stacked on https://github.com/apache/kafka/pull/13442. The first commit does not need to be reviewed separately.)
This PR adds docs for the versioned stores introduced in [KIP-889](https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores). While adding a warning about not changing segment interval between application restarts, I also added the analogous warning to window stores as well.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vcrfxia commented on a diff in pull request #13444: KAFKA-14491: [21/N] Docs updates for versioned state stores
Posted by "vcrfxia (via GitHub)" <gi...@apache.org>.
vcrfxia commented on code in PR #13444:
URL: https://github.com/apache/kafka/pull/13444#discussion_r1154765044
##########
docs/streams/developer-guide/processor-api.html:
##########
@@ -396,12 +400,68 @@ <h2>
</ul>
</p>
</div>
+ <div class="section" id="versioned-state-stores">
+ <span id="streams-developer-guide-state-store-versioned"></span><h3><a class="toc-backref" href="#id12">Versioned Key-Value State Stores</a><a class="headerlink" href="#versioned-state-stores" title="Permalink to this headline"></a></h3>
+ <p>Versioned key-value state stores are available since Kafka Streams 3.5.
+ Rather than storing a single record version (value and timestamp) per key,
+ versioned state stores may store multiple record versions per key. This
+ allows versioned state stores to support timestamped retrieval operations
+ to return the latest record (per key) as of a specified timestamp, which
+ enables use cases such as temporal stream-table joins.</p>
+ <p>You can create a persistent, versioned state store by passing a
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentVersionedKeyValueStore(java.lang.String,java.time.Duration)">VersionedBytesStoreSupplier</a>
+ to the
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#versionedKeyValueStoreBuilder(java.lang.String,java.time.Duration)">versionedKeyValueStoreBuilder</a>,
+ or by implementing your own
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/VersionedKeyValueStore.html">VersionedKeyValueStore</a>.</p>
+ <p>Each versioned store has an associated, fixed-duration <em>history retention</em>
+ parameter which specifies long old record versions should be kept for.
+ In particular, a versioned store guarantees to return accurate results for
+ timestamped retrieval operations where the timestamp being queried is within
+ history retention of the current observed stream time.</p>
+ <p>History retention also doubles as its <em>grace period</em>, which determines
+ how far back in time out-of-order writes to the store will be accepted. A
+ versioned store will not accept writes (inserts, updates, or deletions) if
+ the timestamp associated with the write is older than the current observed
+ stream time by more than the grace period. Stream time in this context is
+ tracked per-partition, rather than per-key, which means it's important
+ that grace period (i.e., history retention) be set high enough to
+ accommodate a record with one key arriving out-of-order relative to a
+ record for another key.</p>
+ <p>Because the memory footprint of versioned key-value stores is higher than
+ that of non-versioned key-value stores, you may want to adjust your
+ <a class="reference internal" href="memory-mgmt.html#streams-developer-guide-memory-management-rocksdb"><span class="std std-ref">RocksDB memory settings</span></a>
+ accordingly. Benchmarking your application with versioned stores is also
+ advised as performance is expected to be worse than when using non-versioned
+ stores.</p>
+ <p>Versioned stores do not support caching or interactive queries at this time.
+ Also, window stores may not be versioned.</p>
+ <b>Upgrade note:</b> Versioned state stores are opt-in only; no automatic
Review Comment:
I've added a section into the upgrade guide which links to the developer guide section. I think this is a good compromise in order to keep the upgrade guide smaller, and to be consistent with the fact that the detailed upgrade note for timestamped key-value stores is also in the developer guide (rather than the upgrade guide).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax merged pull request #13444: KAFKA-14491: [21/N] Docs updates for versioned state stores
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #13444:
URL: https://github.com/apache/kafka/pull/13444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a diff in pull request #13444: KAFKA-14491: [21/N] Docs updates for versioned state stores
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #13444:
URL: https://github.com/apache/kafka/pull/13444#discussion_r1151302961
##########
docs/streams/developer-guide/processor-api.html:
##########
@@ -396,12 +400,68 @@ <h2>
</ul>
</p>
</div>
+ <div class="section" id="versioned-state-stores">
+ <span id="streams-developer-guide-state-store-versioned"></span><h3><a class="toc-backref" href="#id12">Versioned Key-Value State Stores</a><a class="headerlink" href="#versioned-state-stores" title="Permalink to this headline"></a></h3>
+ <p>Versioned key-value state stores are available since Kafka Streams 3.5.
+ Rather than storing a single record version (value and timestamp) per key,
+ versioned state stores may store multiple record versions per key. This
+ allows versioned state stores to support timestamped retrieval operations
+ to return the latest record (per key) as of a specified timestamp, which
+ enables use cases such as temporal stream-table joins.</p>
+ <p>You can create a persistent, versioned state store by passing a
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentVersionedKeyValueStore(java.lang.String,java.time.Duration)">VersionedBytesStoreSupplier</a>
+ to the
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#versionedKeyValueStoreBuilder(java.lang.String,java.time.Duration)">versionedKeyValueStoreBuilder</a>,
+ or by implementing your own
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/VersionedKeyValueStore.html">VersionedKeyValueStore</a>.</p>
+ <p>Each versioned store has an associated, fixed-duration <em>history retention</em>
+ parameter which specifies long old record versions should be kept for.
+ In particular, a versioned store guarantees to return accurate results for
+ timestamped retrieval operations where the timestamp being queried is within
+ history retention of the current observed stream time.</p>
+ <p>History retention also doubles as its <em>grace period</em>, which determines
+ how far back in time out-of-order writes to the store will be accepted. A
+ versioned store will not accept writes (inserts, updates, or deletions) if
+ the timestamp associated with the write is older than the current observed
+ stream time by more than the grace period. Stream time in this context is
+ tracked per-partition, rather than per-key, which means it's important
+ that grace period (i.e., history retention) be set high enough to
+ accommodate a record with one key arriving out-of-order relative to a
+ record for another key.</p>
+ <p>Because the memory footprint of versioned key-value stores is higher than
+ that of non-versioned key-value stores, you may want to adjust your
+ <a class="reference internal" href="memory-mgmt.html#streams-developer-guide-memory-management-rocksdb"><span class="std std-ref">RocksDB memory settings</span></a>
+ accordingly. Benchmarking your application with versioned stores is also
+ advised as performance is expected to be worse than when using non-versioned
+ stores.</p>
+ <p>Versioned stores do not support caching or interactive queries at this time.
+ Also, window stores may not be versioned.</p>
+ <b>Upgrade note:</b> Versioned state stores are opt-in only; no automatic
Review Comment:
Not sure -- we don't have an actually consistent guideline how to organize docs.
If we have it for tkv-store (I did not remember this), it should be fine to add it here, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vcrfxia commented on a diff in pull request #13444: KAFKA-14491: [21/N] Docs updates for versioned state stores
Posted by "vcrfxia (via GitHub)" <gi...@apache.org>.
vcrfxia commented on code in PR #13444:
URL: https://github.com/apache/kafka/pull/13444#discussion_r1154757216
##########
docs/streams/developer-guide/processor-api.html:
##########
@@ -396,12 +400,68 @@ <h2>
</ul>
</p>
</div>
+ <div class="section" id="versioned-state-stores">
+ <span id="streams-developer-guide-state-store-versioned"></span><h3><a class="toc-backref" href="#id12">Versioned Key-Value State Stores</a><a class="headerlink" href="#versioned-state-stores" title="Permalink to this headline"></a></h3>
+ <p>Versioned key-value state stores are available since Kafka Streams 3.5.
+ Rather than storing a single record version (value and timestamp) per key,
+ versioned state stores may store multiple record versions per key. This
+ allows versioned state stores to support timestamped retrieval operations
+ to return the latest record (per key) as of a specified timestamp, which
+ enables use cases such as temporal stream-table joins.</p>
+ <p>You can create a persistent, versioned state store by passing a
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentVersionedKeyValueStore(java.lang.String,java.time.Duration)">VersionedBytesStoreSupplier</a>
+ to the
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#versionedKeyValueStoreBuilder(java.lang.String,java.time.Duration)">versionedKeyValueStoreBuilder</a>,
+ or by implementing your own
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/VersionedKeyValueStore.html">VersionedKeyValueStore</a>.</p>
+ <p>Each versioned store has an associated, fixed-duration <em>history retention</em>
+ parameter which specifies long old record versions should be kept for.
+ In particular, a versioned store guarantees to return accurate results for
+ timestamped retrieval operations where the timestamp being queried is within
+ history retention of the current observed stream time.</p>
+ <p>History retention also doubles as its <em>grace period</em>, which determines
+ how far back in time out-of-order writes to the store will be accepted. A
+ versioned store will not accept writes (inserts, updates, or deletions) if
+ the timestamp associated with the write is older than the current observed
+ stream time by more than the grace period. Stream time in this context is
Review Comment:
Hm, this looks to be inconsistent throughout the existing codebase since searching for both "stream time" and "stream-time" turn up a bunch of results across docs and also javadocs/code comments. Do we have an agreed upon standard for the repo? Personally I prefer "stream time" but unification is probably more important :) If there's already consensus I can make the updates in this PR. Otherwise, might be best left for a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vcrfxia commented on a diff in pull request #13444: KAFKA-14491: [21/N] Docs updates for versioned state stores
Posted by "vcrfxia (via GitHub)" <gi...@apache.org>.
vcrfxia commented on code in PR #13444:
URL: https://github.com/apache/kafka/pull/13444#discussion_r1151114084
##########
docs/streams/developer-guide/processor-api.html:
##########
@@ -396,12 +400,68 @@ <h2>
</ul>
</p>
</div>
+ <div class="section" id="versioned-state-stores">
+ <span id="streams-developer-guide-state-store-versioned"></span><h3><a class="toc-backref" href="#id12">Versioned Key-Value State Stores</a><a class="headerlink" href="#versioned-state-stores" title="Permalink to this headline"></a></h3>
+ <p>Versioned key-value state stores are available since Kafka Streams 3.5.
+ Rather than storing a single record version (value and timestamp) per key,
+ versioned state stores may store multiple record versions per key. This
+ allows versioned state stores to support timestamped retrieval operations
+ to return the latest record (per key) as of a specified timestamp, which
+ enables use cases such as temporal stream-table joins.</p>
+ <p>You can create a persistent, versioned state store by passing a
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentVersionedKeyValueStore(java.lang.String,java.time.Duration)">VersionedBytesStoreSupplier</a>
+ to the
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#versionedKeyValueStoreBuilder(java.lang.String,java.time.Duration)">versionedKeyValueStoreBuilder</a>,
+ or by implementing your own
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/VersionedKeyValueStore.html">VersionedKeyValueStore</a>.</p>
+ <p>Each versioned store has an associated, fixed-duration <em>history retention</em>
+ parameter which specifies long old record versions should be kept for.
+ In particular, a versioned store guarantees to return accurate results for
+ timestamped retrieval operations where the timestamp being queried is within
+ history retention of the current observed stream time.</p>
+ <p>History retention also doubles as its <em>grace period</em>, which determines
+ how far back in time out-of-order writes to the store will be accepted. A
+ versioned store will not accept writes (inserts, updates, or deletions) if
+ the timestamp associated with the write is older than the current observed
+ stream time by more than the grace period. Stream time in this context is
+ tracked per-partition, rather than per-key, which means it's important
+ that grace period (i.e., history retention) be set high enough to
+ accommodate a record with one key arriving out-of-order relative to a
+ record for another key.</p>
+ <p>Because the memory footprint of versioned key-value stores is higher than
+ that of non-versioned key-value stores, you may want to adjust your
+ <a class="reference internal" href="memory-mgmt.html#streams-developer-guide-memory-management-rocksdb"><span class="std std-ref">RocksDB memory settings</span></a>
+ accordingly. Benchmarking your application with versioned stores is also
+ advised as performance is expected to be worse than when using non-versioned
+ stores.</p>
+ <p>Versioned stores do not support caching or interactive queries at this time.
+ Also, window stores may not be versioned.</p>
+ <b>Upgrade note:</b> Versioned state stores are opt-in only; no automatic
Review Comment:
Hm, the reason I had it here is because I saw that we had an analogous note for timestamped stores in these docs above: https://github.com/apache/kafka/blob/5afedd9ac37c4d740f47867cfd31eaed15dc542f/docs/streams/developer-guide/processor-api.html#L387
If you think this type of content belongs in the upgrade guide instead, I will move it over (for versioned stores) and simply link to the upgrade guide from here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vcrfxia commented on a diff in pull request #13444: KAFKA-14491: [21/N] Docs updates for versioned state stores
Posted by "vcrfxia (via GitHub)" <gi...@apache.org>.
vcrfxia commented on code in PR #13444:
URL: https://github.com/apache/kafka/pull/13444#discussion_r1154755917
##########
docs/streams/developer-guide/processor-api.html:
##########
@@ -261,11 +262,13 @@ <h2>
space.</li>
<li>RocksDB settings can be fine-tuned, see
<a class="reference internal" href="config-streams.html#streams-developer-guide-rocksdb-config"><span class="std std-ref">RocksDB configuration</span></a>.</li>
- <li>Available <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentKeyValueStore-java.lang.String-">store variants</a>:
- time window key-value store, session window key-value store.</li>
- <li>Use <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentTimestampedKeyValueStore-java.lang.String-">persistentTimestampedKeyValueStore</a>
- when you need a persistent key-(value/timestamp) store that supports put/get/delete and range queries.</li>
- <li>Use <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentTimestampedWindowStore-java.lang.String-java.time.Duration-java.time.Duration-boolean-">persistentTimestampedWindowStore</a>
+ <li>Available <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentKeyValueStore(java.lang.String)">store variants</a>:
+ versioned key-value store, time window key-value store, session window key-value store.</li>
Review Comment:
Added timestamped KV, regular window stores (previously only timestamped window stores were represented), and session stores into this section, in order to be comprehensive. LMK if it's too much now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on pull request #13444: KAFKA-14491: [21/N] Docs updates for versioned state stores
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #13444:
URL: https://github.com/apache/kafka/pull/13444#issuecomment-1499924925
We should also add a note to the top level upgrade guide `(ie, `docs/upgrade.html`); KIP-889 is a major feature that should be mentioned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vcrfxia commented on pull request #13444: KAFKA-14491: [21/N] Docs updates for versioned state stores
Posted by "vcrfxia (via GitHub)" <gi...@apache.org>.
vcrfxia commented on PR #13444:
URL: https://github.com/apache/kafka/pull/13444#issuecomment-1504673370
Sorry for the delay in responding -- just updated the PR with your latest suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mjsax commented on a diff in pull request #13444: KAFKA-14491: [21/N] Docs updates for versioned state stores
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #13444:
URL: https://github.com/apache/kafka/pull/13444#discussion_r1150042983
##########
docs/streams/developer-guide/processor-api.html:
##########
@@ -261,11 +262,13 @@ <h2>
space.</li>
<li>RocksDB settings can be fine-tuned, see
<a class="reference internal" href="config-streams.html#streams-developer-guide-rocksdb-config"><span class="std std-ref">RocksDB configuration</span></a>.</li>
- <li>Available <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentKeyValueStore-java.lang.String-">store variants</a>:
- time window key-value store, session window key-value store.</li>
- <li>Use <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentTimestampedKeyValueStore-java.lang.String-">persistentTimestampedKeyValueStore</a>
- when you need a persistent key-(value/timestamp) store that supports put/get/delete and range queries.</li>
- <li>Use <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentTimestampedWindowStore-java.lang.String-java.time.Duration-java.time.Duration-boolean-">persistentTimestampedWindowStore</a>
+ <li>Available <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentKeyValueStore(java.lang.String)">store variants</a>:
+ versioned key-value store, time window key-value store, session window key-value store.</li>
Review Comment:
This part of the docs seems to be messy? We don't list timestampe-kv here but as own bullet point below. Below, session-store is missing as bullet point...
##########
docs/streams/developer-guide/processor-api.html:
##########
@@ -396,12 +400,68 @@ <h2>
</ul>
</p>
</div>
+ <div class="section" id="versioned-state-stores">
+ <span id="streams-developer-guide-state-store-versioned"></span><h3><a class="toc-backref" href="#id12">Versioned Key-Value State Stores</a><a class="headerlink" href="#versioned-state-stores" title="Permalink to this headline"></a></h3>
+ <p>Versioned key-value state stores are available since Kafka Streams 3.5.
+ Rather than storing a single record version (value and timestamp) per key,
+ versioned state stores may store multiple record versions per key. This
+ allows versioned state stores to support timestamped retrieval operations
+ to return the latest record (per key) as of a specified timestamp, which
+ enables use cases such as temporal stream-table joins.</p>
Review Comment:
`which enables use cases such as temporal stream-table joins.` -- I would remove this -- we have s-t-join already, so "enables" sounds weird. -- We also don't need to "justify/explain" why we have a store in the docs.
##########
docs/streams/developer-guide/processor-api.html:
##########
@@ -396,12 +400,68 @@ <h2>
</ul>
</p>
</div>
+ <div class="section" id="versioned-state-stores">
+ <span id="streams-developer-guide-state-store-versioned"></span><h3><a class="toc-backref" href="#id12">Versioned Key-Value State Stores</a><a class="headerlink" href="#versioned-state-stores" title="Permalink to this headline"></a></h3>
+ <p>Versioned key-value state stores are available since Kafka Streams 3.5.
+ Rather than storing a single record version (value and timestamp) per key,
+ versioned state stores may store multiple record versions per key. This
+ allows versioned state stores to support timestamped retrieval operations
+ to return the latest record (per key) as of a specified timestamp, which
+ enables use cases such as temporal stream-table joins.</p>
+ <p>You can create a persistent, versioned state store by passing a
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentVersionedKeyValueStore(java.lang.String,java.time.Duration)">VersionedBytesStoreSupplier</a>
+ to the
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#versionedKeyValueStoreBuilder(java.lang.String,java.time.Duration)">versionedKeyValueStoreBuilder</a>,
+ or by implementing your own
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/VersionedKeyValueStore.html">VersionedKeyValueStore</a>.</p>
+ <p>Each versioned store has an associated, fixed-duration <em>history retention</em>
+ parameter which specifies long old record versions should be kept for.
+ In particular, a versioned store guarantees to return accurate results for
+ timestamped retrieval operations where the timestamp being queried is within
+ history retention of the current observed stream time.</p>
+ <p>History retention also doubles as its <em>grace period</em>, which determines
+ how far back in time out-of-order writes to the store will be accepted. A
+ versioned store will not accept writes (inserts, updates, or deletions) if
+ the timestamp associated with the write is older than the current observed
+ stream time by more than the grace period. Stream time in this context is
+ tracked per-partition, rather than per-key, which means it's important
+ that grace period (i.e., history retention) be set high enough to
+ accommodate a record with one key arriving out-of-order relative to a
+ record for another key.</p>
+ <p>Because the memory footprint of versioned key-value stores is higher than
+ that of non-versioned key-value stores, you may want to adjust your
+ <a class="reference internal" href="memory-mgmt.html#streams-developer-guide-memory-management-rocksdb"><span class="std std-ref">RocksDB memory settings</span></a>
+ accordingly. Benchmarking your application with versioned stores is also
+ advised as performance is expected to be worse than when using non-versioned
+ stores.</p>
+ <p>Versioned stores do not support caching or interactive queries at this time.
+ Also, window stores may not be versioned.</p>
+ <b>Upgrade note:</b> Versioned state stores are opt-in only; no automatic
Review Comment:
Should we just refer to the upgrade guide? Or maybe just omit it here at all?
##########
docs/streams/developer-guide/processor-api.html:
##########
@@ -396,12 +400,68 @@ <h2>
</ul>
</p>
</div>
+ <div class="section" id="versioned-state-stores">
+ <span id="streams-developer-guide-state-store-versioned"></span><h3><a class="toc-backref" href="#id12">Versioned Key-Value State Stores</a><a class="headerlink" href="#versioned-state-stores" title="Permalink to this headline"></a></h3>
+ <p>Versioned key-value state stores are available since Kafka Streams 3.5.
+ Rather than storing a single record version (value and timestamp) per key,
+ versioned state stores may store multiple record versions per key. This
+ allows versioned state stores to support timestamped retrieval operations
+ to return the latest record (per key) as of a specified timestamp, which
+ enables use cases such as temporal stream-table joins.</p>
+ <p>You can create a persistent, versioned state store by passing a
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#persistentVersionedKeyValueStore(java.lang.String,java.time.Duration)">VersionedBytesStoreSupplier</a>
+ to the
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/Stores.html#versionedKeyValueStoreBuilder(java.lang.String,java.time.Duration)">versionedKeyValueStoreBuilder</a>,
+ or by implementing your own
+ <a class="reference external" href="/{{version}}/javadoc/org/apache/kafka/streams/state/VersionedKeyValueStore.html">VersionedKeyValueStore</a>.</p>
+ <p>Each versioned store has an associated, fixed-duration <em>history retention</em>
+ parameter which specifies long old record versions should be kept for.
+ In particular, a versioned store guarantees to return accurate results for
+ timestamped retrieval operations where the timestamp being queried is within
+ history retention of the current observed stream time.</p>
+ <p>History retention also doubles as its <em>grace period</em>, which determines
+ how far back in time out-of-order writes to the store will be accepted. A
+ versioned store will not accept writes (inserts, updates, or deletions) if
+ the timestamp associated with the write is older than the current observed
+ stream time by more than the grace period. Stream time in this context is
Review Comment:
`stream time` -> `stream-time` (multiple times)
##########
streams/src/main/java/org/apache/kafka/streams/state/Stores.java:
##########
@@ -166,6 +248,11 @@ public String metricsScope() {
* This store supplier can be passed into a {@link #windowStoreBuilder(WindowBytesStoreSupplier, Serde, Serde)}.
* If you want to create a {@link TimestampedWindowStore} you should use
* {@link #persistentTimestampedWindowStore(String, Duration, Duration, boolean)} to create a store supplier instead.
+ * <p>
+ * Note that it is not safe to change the value of {@code retentionPeriod} between
Review Comment:
I think we can give it a try as you propose -- docs are "easy" to change, so I would not worry about it too much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vcrfxia commented on a diff in pull request #13444: KAFKA-14491: [21/N] Docs updates for versioned state stores
Posted by "vcrfxia (via GitHub)" <gi...@apache.org>.
vcrfxia commented on code in PR #13444:
URL: https://github.com/apache/kafka/pull/13444#discussion_r1147020240
##########
streams/src/main/java/org/apache/kafka/streams/state/Stores.java:
##########
@@ -166,6 +248,11 @@ public String metricsScope() {
* This store supplier can be passed into a {@link #windowStoreBuilder(WindowBytesStoreSupplier, Serde, Serde)}.
* If you want to create a {@link TimestampedWindowStore} you should use
* {@link #persistentTimestampedWindowStore(String, Duration, Duration, boolean)} to create a store supplier instead.
+ * <p>
+ * Note that it is not safe to change the value of {@code retentionPeriod} between
Review Comment:
I worry slightly that adding this warning about not changing retention period between restarts will give the false impression that it is OK to change, for example, window size between restarts. While I don't think that's a good idea, I also don't think anything necessarily breaks if a user does that (unlike for history retention where something definitely could break, if segment interval is changed under the hood as well) so it's unclear whether that should be discouraged in the docs or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org