You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/05/06 02:40:51 UTC
[GitHub] [kafka] abbccdda commented on a change in pull request #8621: KAFKA-9466: Update Kafka Streams docs for KIP-447
abbccdda commented on a change in pull request #8621:
URL: https://github.com/apache/kafka/pull/8621#discussion_r420514826
##########
File path: docs/streams/upgrade-guide.html
##########
@@ -52,6 +52,20 @@ <h1>Upgrade Guide and API Changes</h1>
<li> restart all new ({{fullDotVersion}}) application instances </li>
</ul>
+ <p>
+ As of Kafka Streams 2.6.x a new processing mode <code>"exactly_once_beta"</code> (configurable via parameter
+ <code>processing.guarantee</code>) is available.
+ To use this new feature, your brokers must be on version 2.5.x or newer.
+ A switch from <code>"exactly_once"</code> to <code>"exactly_once_beta"</code> (or the other way around) is
+ only possible if the application is on version 2.6.x.
+ Hence, if you want to upgrade your application from an older version and enable this feature,
+ you first need to upgrade your application to version 2.6.x staying on <code>"exactly_once"</code>
+ and afterwards do second round of rolling bounces to switch to <code>"exactly_once_beta"</code>.
+ For a downgrade do the revers: first switch the config from <code>"exactly_once_beta"</code> to
+ <code>"exactly_once"</code>to disable the feature on your 2.6.x application.
+ Afterward, you can downgrade you application to a pre 2.6.x version.
Review comment:
downgrade your
##########
File path: docs/streams/upgrade-guide.html
##########
@@ -72,6 +86,15 @@ <h1>Upgrade Guide and API Changes</h1>
More details about the new config <code>StreamsConfig#TOPOLOGY_OPTIMIZATION</code> can be found in <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-295%3A+Add+Streams+Configuration+Allowing+for+Optional+Topology+Optimization">KIP-295</a>.
</p>
+ <h3><a id="streams_api_changes_260" href="#streams_api_changes_260">Streams API changes in 2.6.0</a></h3>
+ <p>
+ We added a new processing mode that improves application scalability using exactly-once guarantees
+ (via <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics">KIP-447</a>>).
+ You can enable this new feature be setting the configuration parameter <code>processing.guarantee</code> to the
+ new value <code>"exactly_once_beta"</code>.
+ Note that you need brokers with version 2.5 or newer to use this new feature.
Review comment:
nit: could just say `this feature` as we mentioned new feature earlier.
##########
File path: docs/streams/core-concepts.html
##########
@@ -206,16 +206,16 @@ <h2><a id="streams_processing_guarantee" href="#streams_processing_guarantee">Pr
to the stream processing pipeline, known as the <a href="http://lambda-architecture.net/">Lambda Architecture</a>.
Prior to 0.11.0.0, Kafka only provides at-least-once delivery guarantees and hence any stream processing systems that leverage it as the backend storage could not guarantee end-to-end exactly-once semantics.
In fact, even for those stream processing systems that claim to support exactly-once processing, as long as they are reading from / writing to Kafka as the source / sink, their applications cannot actually guarantee that
- no duplicates will be generated throughout the pipeline.
+ no duplicates will be generated throughout the pipeline.<br />
Since the 0.11.0.0 release, Kafka has added support to allow its producers to send messages to different topic partitions in a <a href="https://kafka.apache.org/documentation/#semantics">transactional and idempotent manner</a>,
and Kafka Streams has hence added the end-to-end exactly-once processing semantics by leveraging these features.
More specifically, it guarantees that for any record read from the source Kafka topics, its processing results will be reflected exactly once in the output Kafka topic as well as in the state stores for stateful operations.
Note the key difference between Kafka Streams end-to-end exactly-once guarantee with other stream processing frameworks' claimed guarantees is that Kafka Streams tightly integrates with the underlying Kafka storage system and ensure that
commits on the input topic offsets, updates on the state stores, and writes to the output topics will be completed atomically instead of treating Kafka as an external system that may have side-effects.
- To read more details on how this is done inside Kafka Streams, readers are recommended to read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-129%3A+Streams+Exactly-Once+Semantics">KIP-129</a>.
+ To read more details on how this is done inside Kafka Streams, readers are recommended to read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-129%3A+Streams+Exactly-Once+Semantics">KIP-129</a>.<br />
- In order to achieve exactly-once semantics when running Kafka Streams applications, users can simply set the <code>processing.guarantee</code> config value to <b>exactly_once</b> (default value is <b>at_least_once</b>).
+ In order to achieve exactly-once semantics when running Kafka Streams applications, users can simply set the <code>processing.guarantee</code> config value (default value is <b>at_least_once</b>) to <b>exactly_once</b> or <b>exactly_once_beta</b> (requires brokers version 2.5 or newer).
More details can be found in the <a href="/{{version}}/documentation#streamsconfigs"><b>Kafka Streams Configs</b></a> section.
Review comment:
It's a bit unfortunate that inside StreamsConfig we don't have further information concerning the difference between EOS and EOS beta. Maybe we just talk more into details to compare between the two here? Just need to let user understand that EOS beta does guarantee the same semantics with potentially better scalability as an improvement in 2.5 & 2.6.
##########
File path: docs/upgrade.html
##########
@@ -19,6 +19,12 @@
<script id="upgrade-template" type="text/x-handlebars-template">
+<h5><a id="upgrade_260_notable" href="#upgrade_260_notable">Notable changes in 2.6.0</a></h5>
+<ul>
+ <li>Kafka Streams adds a new processing mode (requires broker 2.5 or newer) that improves application
Review comment:
Do we need to attach a link to KIP here as well?
##########
File path: docs/streams/upgrade-guide.html
##########
@@ -72,6 +86,15 @@ <h1>Upgrade Guide and API Changes</h1>
More details about the new config <code>StreamsConfig#TOPOLOGY_OPTIMIZATION</code> can be found in <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-295%3A+Add+Streams+Configuration+Allowing+for+Optional+Topology+Optimization">KIP-295</a>.
</p>
+ <h3><a id="streams_api_changes_260" href="#streams_api_changes_260">Streams API changes in 2.6.0</a></h3>
+ <p>
+ We added a new processing mode that improves application scalability using exactly-once guarantees
+ (via <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-447%3A+Producer+scalability+for+exactly+once+semantics">KIP-447</a>>).
+ You can enable this new feature be setting the configuration parameter <code>processing.guarantee</code> to the
Review comment:
by setting
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org