You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/04/21 20:00:15 UTC
[GitHub] [kafka] mjsax commented on a change in pull request #10573: KAFKA-12574: KIP-732, Deprecate eos-alpha and replace eos-beta with eos-v2
mjsax commented on a change in pull request #10573:
URL: https://github.com/apache/kafka/pull/10573#discussion_r617826977
##########
File path: clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java
##########
@@ -642,9 +647,13 @@ public void beginTransaction() throws ProducerFencedException {
* to the partition leader. See the exception for more details
* @throws KafkaException if the producer has encountered a previous fatal or abortable error, or for any
* other unexpected error
+ *
+ * @deprecated Since 3.0.0, will be removed in 4.0. Use {@link #sendOffsetsToTransaction(Map, ConsumerGroupMetadata)} instead.
*/
+ @Deprecated
public void sendOffsetsToTransaction(Map<TopicPartition, OffsetAndMetadata> offsets,
String consumerGroupId) throws ProducerFencedException {
+ log.warn("This method has been deprecated and will be removed in 4.0, please use #sendOffsetsToTransaction(Map, ConsumerGroupMetadata) instead");
Review comment:
Do we really need to log a WAR? We never did anything link this in KS code base in the past. Marking the method as `@Deprecated` should be sufficient IMHO? Or it such a WARN log custom in client code base?
##########
File path: clients/src/main/java/org/apache/kafka/clients/producer/Producer.java
##########
@@ -49,7 +49,10 @@
/**
* See {@link KafkaProducer#sendOffsetsToTransaction(Map, String)}
+ *
+
Review comment:
nit: needs cleanup
##########
File path: docs/streams/core-concepts.html
##########
@@ -291,16 +291,18 @@ <h2 class="anchor-heading"><a id="streams_processing_guarantee" class="anchor-li
commits on the input topic offsets, updates on the state stores, and writes to the output topics will be completed atomically instead of treating Kafka as an external system that may have side-effects.
For more information on how this is done inside Kafka Streams, see <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-129%3A+Streams+Exactly-Once+Semantics">KIP-129</a>.<br />
- As of the 2.6.0 release, Kafka Streams supports an improved implementation of exactly-once processing, named "exactly-once beta",
+ As of the 2.6.0 release, Kafka Streams supports an improved implementation of exactly-once processing, named "exactly-once v2",
which requires broker version 2.5.0 or newer.
This implementation is more efficient, because it reduces client and broker resource utilization, like client threads and used network connections,
and it enables higher throughput and improved scalability.
+ As of the 3.0.0 release, the old "alpha" version of exactly-once has been deprecated. Users are encouraged to use exactly-once v2 for
Review comment:
We never used "alpha" anywhere in public, so might be better to avoid this term at all?
`old "alpha" -> "first"
##########
File path: docs/streams/developer-guide/config-streams.html
##########
@@ -667,12 +668,14 @@ <h4><a class="toc-backref" href="#id30">probing.rebalance.interval.ms</a><a clas
<span id="streams-developer-guide-processing-guarantee"></span><h4><a class="toc-backref" href="#id25">processing.guarantee</a><a class="headerlink" href="#processing-guarantee" title="Permalink to this headline"></a></h4>
<blockquote>
<div>The processing guarantee that should be used.
- Possible values are <code class="docutils literal"><span class="pre">"at_least_once"</span></code> (default),
- <code class="docutils literal"><span class="pre">"exactly_once"</span></code> (for EOS version 1),
- and <code class="docutils literal"><span class="pre">"exactly_once_beta"</span></code> (for EOS version 2).
- Using <code class="docutils literal"><span class="pre">"exactly_once"</span></code> requires broker
- version 0.11.0 or newer, while using <code class="docutils literal"><span class="pre">"exactly_once_beta"</span></code>
- requires broker version 2.5 or newer.
+ Possible values are <code class="docutils literal"><span class="pre">"at_least_once"</span></code> (default)
+ and <code class="docutils literal"><span class="pre">"exactly_once_v2"</span></code> (for EOS version 2).
+ Deprecated config options are <code class="docutils literal"><span class="pre">"exactly_once"</span></code> (for EOS alpha),
Review comment:
Do we need to elaborate on all deprecated configs?
##########
File path: docs/streams/upgrade-guide.html
##########
@@ -93,6 +95,12 @@ <h1>Upgrade Guide and API Changes</h1>
</p>
<h3><a id="streams_api_changes_300" href="#streams_api_changes_300">Streams API changes in 3.0.0</a></h3>
+ <p>
+ The <code>StreamsConfig.EXACTLY_ONCE</code> and <code>StreamsConfig.EXACTLY_ONCE_BETA</code> configs have been deprecated, and a new <code>StreamsConfig.EXACTLY_ONCE_V2</code> config has been
Review comment:
Might be good to highlight at `beta -> v2` is just a renaming ?
##########
File path: docs/streams/upgrade-guide.html
##########
@@ -53,17 +53,19 @@ <h1>Upgrade Guide and API Changes</h1>
</ul>
<p>
- Starting in Kafka Streams 2.6.x, a new processing mode is available, named EOS version 2, which is configurable by setting
- <code>processing.guarantee</code> to <code>"exactly_once_beta"</code>.
- <b>NOTE:</b> The <code>"exactly_once_beta"</code> processing mode is ready for production (<i>i.e.</i>, it's not "beta" software).
+ Starting in Kafka Streams 2.6.x, a new processing mode is available, named EOS version 2. This can be configured
+ by setting <code>StreamsConfig.PROCESSING_GUARANTEE</code> to <code>StreamsConfig.EXACTLY_ONCE_V2</code> for
+ application versions 3.0+, or setting it to <code>StreamsConfig.EXACTLY_ONCE_BETA</code> for versions between 2.6 and 3.0.
To use this new feature, your brokers must be on version 2.5.x or newer.
- A switch from <code>"exactly_once"</code> to <code>"exactly_once_beta"</code> (or the other way around) is
- only possible if the application is on version 2.6.x.
- If you want to upgrade your application from an older version and enable this feature,
- you first need to upgrade your application to version 2.6.x, staying on <code>"exactly_once"</code>,
- and then do second round of rolling bounces to switch to <code>"exactly_once_beta"</code>.
- For a downgrade, do the reverse: first switch the config from <code>"exactly_once_beta"</code> to
- <code>"exactly_once"</code> to disable the feature in your 2.6.x application.
+ If you want to upgrade your EOS application from an older version and enable this feature,
+ you first need to upgrade your application to version 2.6.x, staying on <code>StreamsConfig.EXACTLY_ONCE</code>,
Review comment:
`you first need to upgrade your application to version 2.6.x`
Why would that be required? On can go to `3.0` directly from my understanding?
##########
File path: docs/streams/developer-guide/config-streams.html
##########
@@ -293,8 +293,9 @@ <h4><a class="toc-backref" href="#id5">bootstrap.servers</a><a class="headerlink
</tr>
<tr class="row-even"><td>processing.guarantee</td>
<td>Medium</td>
- <td colspan="2">The processing mode. Can be either <code class="docutils literal"><span class="pre">"at_least_once"</span></code> (default),
- <code class="docutils literal"><span class="pre">"exactly_once"</span></code> (for EOS version 1), or <code class="docutils literal"><span class="pre">"exactly_once_beta"</span></code> (for EOS version 2)</td>.
+ <td colspan="2">The processing mode. Can be either <code class="docutils literal"><span class="pre">"at_least_once"</span></code> (default)
+ or <code class="docutils literal"><span class="pre">"exactly_once_v2"</span></code> (for EOS version 2). Deprecated config options are
Review comment:
We should add `requires broker version 2.5 or newer`
##########
File path: docs/streams/developer-guide/config-streams.html
##########
@@ -293,8 +293,9 @@ <h4><a class="toc-backref" href="#id5">bootstrap.servers</a><a class="headerlink
</tr>
<tr class="row-even"><td>processing.guarantee</td>
<td>Medium</td>
- <td colspan="2">The processing mode. Can be either <code class="docutils literal"><span class="pre">"at_least_once"</span></code> (default),
- <code class="docutils literal"><span class="pre">"exactly_once"</span></code> (for EOS version 1), or <code class="docutils literal"><span class="pre">"exactly_once_beta"</span></code> (for EOS version 2)</td>.
+ <td colspan="2">The processing mode. Can be either <code class="docutils literal"><span class="pre">"at_least_once"</span></code> (default)
+ or <code class="docutils literal"><span class="pre">"exactly_once_v2"</span></code> (for EOS version 2). Deprecated config options are
+ <code class="docutils literal"><span class="pre">"exactly_once"</span></code> (for EOS alpha) and <code class="docutils literal"><span class="pre">"exactly_once_beta"</span></code> (for EOS beta)</td>.
Review comment:
don't use `alpha`; maybe just say `(requires broker version 0.11 or newer)
##########
File path: streams/src/main/java/org/apache/kafka/streams/StreamsConfig.java
##########
@@ -1010,18 +1026,51 @@ public StreamsConfig(final Map<?, ?> props) {
protected StreamsConfig(final Map<?, ?> props,
final boolean doLog) {
super(CONFIG, props, doLog);
- eosEnabled = StreamThread.eosEnabled(this);
+ eosEnabled = eosEnabled();
+
+ final String processingModeConfig = getString(StreamsConfig.PROCESSING_GUARANTEE_CONFIG);
+ if (processingModeConfig.equals(EXACTLY_ONCE)) {
+ log.warn("Configuration parameter `{}` is deprecated and will be removed in the 4.0.0 release. " +
+ "Please use `{}` instead. Note that this requires broker version 2.5+ so you should prepare "
+ + "to upgrade your brokers if necessary.", EXACTLY_ONCE, EXACTLY_ONCE_V2);
+ }
+ if (processingModeConfig.equals(EXACTLY_ONCE_BETA)) {
+ log.warn("Configuration parameter `{}` is deprecated and will be removed in the 4.0.0 release. " +
+ "Please use `{}` instead.", EXACTLY_ONCE_BETA, EXACTLY_ONCE_V2);
+ }
+
if (props.containsKey(RETRIES_CONFIG)) {
- log.warn("Configuration parameter `{}` is deprecated and will be removed in 3.0.0 release.", RETRIES_CONFIG);
+ log.warn("Configuration parameter `{}` is deprecated and will be removed in the 4.0.0 release.", RETRIES_CONFIG);
+ }
+ }
+
+ public ProcessingMode processingMode() {
+ if (EXACTLY_ONCE.equals(getString(StreamsConfig.PROCESSING_GUARANTEE_CONFIG))) {
+ return StreamThread.ProcessingMode.EXACTLY_ONCE_ALPHA;
+ } else if (EXACTLY_ONCE_BETA.equals(getString(StreamsConfig.PROCESSING_GUARANTEE_CONFIG))) {
+ return StreamThread.ProcessingMode.EXACTLY_ONCE_V2;
+ } else if (EXACTLY_ONCE_V2.equals(getString(StreamsConfig.PROCESSING_GUARANTEE_CONFIG))) {
+ return StreamThread.ProcessingMode.EXACTLY_ONCE_V2;
+ } else {
+ return StreamThread.ProcessingMode.AT_LEAST_ONCE;
}
}
+ public boolean eosEnabled() {
Review comment:
This is a public API change -- to avoid this (ie also "leaks" and users won't need this method) we originally added it to `StreamThread` and not to `StreamsConfig` (even if it might be "cleaner" to have it on `StreamsConfig`)...
As an alternative, we would introduce `class InternalStreamsConfig extends StreamsConfig`, add the method their.... (If we do this, might be worth to split out a refactoring PR and not piggy-back on the KIP PR...)
Same for the other method below.
##########
File path: docs/streams/upgrade-guide.html
##########
@@ -53,17 +53,19 @@ <h1>Upgrade Guide and API Changes</h1>
</ul>
<p>
- Starting in Kafka Streams 2.6.x, a new processing mode is available, named EOS version 2, which is configurable by setting
- <code>processing.guarantee</code> to <code>"exactly_once_beta"</code>.
- <b>NOTE:</b> The <code>"exactly_once_beta"</code> processing mode is ready for production (<i>i.e.</i>, it's not "beta" software).
+ Starting in Kafka Streams 2.6.x, a new processing mode is available, named EOS version 2. This can be configured
+ by setting <code>StreamsConfig.PROCESSING_GUARANTEE</code> to <code>StreamsConfig.EXACTLY_ONCE_V2</code> for
+ application versions 3.0+, or setting it to <code>StreamsConfig.EXACTLY_ONCE_BETA</code> for versions between 2.6 and 3.0.
To use this new feature, your brokers must be on version 2.5.x or newer.
- A switch from <code>"exactly_once"</code> to <code>"exactly_once_beta"</code> (or the other way around) is
- only possible if the application is on version 2.6.x.
- If you want to upgrade your application from an older version and enable this feature,
- you first need to upgrade your application to version 2.6.x, staying on <code>"exactly_once"</code>,
- and then do second round of rolling bounces to switch to <code>"exactly_once_beta"</code>.
- For a downgrade, do the reverse: first switch the config from <code>"exactly_once_beta"</code> to
- <code>"exactly_once"</code> to disable the feature in your 2.6.x application.
+ If you want to upgrade your EOS application from an older version and enable this feature,
+ you first need to upgrade your application to version 2.6.x, staying on <code>StreamsConfig.EXACTLY_ONCE</code>,
+ and then do second round of rolling bounces to switch to <code>StreamsConfig.EXACTLY_ONCE_BETA</code>. If you
Review comment:
`EXACTLY_ONCE_BETA` or `EXACTLY_ONCE_V2`?
##########
File path: docs/streams/upgrade-guide.html
##########
@@ -53,17 +53,19 @@ <h1>Upgrade Guide and API Changes</h1>
</ul>
<p>
- Starting in Kafka Streams 2.6.x, a new processing mode is available, named EOS version 2, which is configurable by setting
- <code>processing.guarantee</code> to <code>"exactly_once_beta"</code>.
- <b>NOTE:</b> The <code>"exactly_once_beta"</code> processing mode is ready for production (<i>i.e.</i>, it's not "beta" software).
+ Starting in Kafka Streams 2.6.x, a new processing mode is available, named EOS version 2. This can be configured
+ by setting <code>StreamsConfig.PROCESSING_GUARANTEE</code> to <code>StreamsConfig.EXACTLY_ONCE_V2</code> for
+ application versions 3.0+, or setting it to <code>StreamsConfig.EXACTLY_ONCE_BETA</code> for versions between 2.6 and 3.0.
Review comment:
`between 2.6 and 2.8` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org