You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kafka.apache.org by li...@apache.org on 2025/02/13 16:52:10 UTC
(kafka) branch trunk updated: KAFKA-17298: Update upgrade notes for 4.0 KIP-848 (#18756)
This is an automated email from the ASF dual-hosted git repository.
lianetm pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git
The following commit(s) were added to refs/heads/trunk by this push:
new c465cf6b4be KAFKA-17298: Update upgrade notes for 4.0 KIP-848 (#18756)
c465cf6b4be is described below
commit c465cf6b4befc3c29aa8c56804d8eebfb3fb74cf
Author: Lianet Magrans <98...@users.noreply.github.com>
AuthorDate: Thu Feb 13 11:51:56 2025 -0500
KAFKA-17298: Update upgrade notes for 4.0 KIP-848 (#18756)
Reviewers: David Jacot <dj...@confluent.io>
---
.../kafka/clients/consumer/ConsumerConfig.java | 6 +-
docs/ops.html | 75 ++++++++++++++++++++++
docs/upgrade.html | 17 ++++-
3 files changed, 94 insertions(+), 4 deletions(-)
diff --git a/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java b/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java
index b0b39274075..d9cee3ec179 100644
--- a/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java
+++ b/clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java
@@ -121,9 +121,9 @@ public class ConsumerConfig extends AbstractConfig {
*/
public static final String GROUP_REMOTE_ASSIGNOR_CONFIG = "group.remote.assignor";
public static final String DEFAULT_GROUP_REMOTE_ASSIGNOR = null;
- public static final String GROUP_REMOTE_ASSIGNOR_DOC = "The server-side assignor to use. If no assignor is specified, " +
- "the group coordinator will pick one. This configuration is applied only if <code>group.protocol</code> is " +
- "set to \"consumer\".";
+ public static final String GROUP_REMOTE_ASSIGNOR_DOC = "The name of the server-side assignor to use. " +
+ "If not specified, the group coordinator will pick the first assignor defined in the broker config group.consumer.assignors." +
+ "This configuration is applied only if <code>group.protocol</code> is set to \"consumer\".";
/**
* <code>bootstrap.servers</code>
diff --git a/docs/ops.html b/docs/ops.html
index 4827b4e0075..22060e67c75 100644
--- a/docs/ops.html
+++ b/docs/ops.html
@@ -4056,6 +4056,81 @@ $ bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:
<p>For more information, please check <a href="https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+GA+Release+Notes">Kafka Tiered Storage GA Release Notes</a>.
</p>
+ <h3 class="anchor-heading"><a id="consumer_rebalance_protocol" class="anchor-link"></a><a href="#consumer_rebalance_protocol">6.10 Consumer Rebalance Protocol</a></h3>
+
+ <h4 class="anchor-heading"><a id="consumer_rebalance_protocol_overview" class="anchor-link"></a><a href="#consumer_rebalance_protocol_overview">Overview</a></h4>
+
+ <p>Starting from Apache Kafka 4.0, the Next Generation of the Consumer Rebalance Protocol (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol">KIP-848</a>)
+ is Generally Available (GA). It improves the scalability of consumer groups while simplifying consumers. It also decreases rebalance times, thanks to
+ its fully incremental design, which no longer relies on a global synchronization barrier.</p>
+
+ <p>Consumer Groups using the new protocol are now referred to as <code>Consumer</code> groups, while groups using the old protocol are referred to as <code>Classic</code> groups. Note
+ that Classic groups can still be used to form consumer groups using the old protocol.</p>
+
+ <h4 class="anchor-heading"><a id="consumer_rebalance_protocol_server" class="anchor-link"></a><a href="#consumer_rebalance_protocol_server">Server</a></h4>
+
+ <p>The new consumer protocol is automatically enabled on the server since Apache Kafka 4.0. Enabling and disabling the protocol is controlled by the
+ <code>group.version</code> feature flag.</p>
+
+ <p>The consumer heartbeat interval and the session timeout are controlled by the server now with the following configs:</p>
+ <ul>
+ <li><code>group.consumer.heartbeat.interval.ms</code></li>
+ <li><code>group.consumer.session.timeout.ms</code></li>
+ </ul>
+
+ <p>The assignment strategy is also controlled by the server. The <code>group.consumer.assignors</code> configuration can be used to specify the list of available
+ assignors for <code>Consumer</code> groups. By default, the <code>uniform</code> assignor and the <code>range</code> assignor are configured. The first assignor
+ in the list is used by default unless the Consumer selects a different one. It is also possible to implement custom assignment strategies on the server side
+ by implementing the <code>org.apache.kafka.coordinator.group.api.assignor.ConsumerGroupPartitionAssignor</code> interface and specifying the full class name in the configuration.</p>
+
+ <h4 class="anchor-heading"><a id="consumer_rebalance_protocol_consumer" class="anchor-link"></a><a href="#consumer_rebalance_protocol_consumer">Consumer</a></h4>
+
+ <p>Since Apache Kafka 4.0, the Consumer supports the new consumer rebalance protocol. However, the protocol is not enabled by default. The <code>group.protocol</code>
+ configuration must be set to <code>consumer</code> to enable it. When enabled, the new consumer protocol is used alongside an improved threading model.</p>
+
+ <p>The <code>group.remote.assignor</code> configuration is introduced as an optional configuration to overwrite the default assignment strategy configured
+ on the server side.</p>
+
+ <p>The <code>subscribe(SubscriptionPattern)</code> and <code>subscribe(SubscriptionPattern, ConsumerRebalanceListener)</code> methods have been added to
+ subscribe to a regular expression with the new consumer rebalance protocol. With these methods, the regular expression uses the RE2J format and is now evaluated
+ on the server side.</p>
+
+ <p>New metrics have been added to the Consumer when using the new rebalance protocol, mainly providing visibility over the improved threading model.
+ See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1068%3A+New+metrics+for+the+new+KafkaConsumer">New Consumer Metrics</a>.</p>
+
+ <p>When the new rebalance protocol is enabled, the following configurations and APIs are no longer usable:</p>
+ <ul>
+ <li><code>heartbeat.interval.ms</code></li>
+ <li><code>session.timeout.ms</code></li>
+ <li><code>partition.assignment.strategy</code></li>
+ <li><code>enforceRebalance(String)</code> and <code>enforceRebalance()</code></li>
+ </ul>
+
+ <h4 class="anchor-heading"><a id="consumer_rebalance_protocol_upgrade" class="anchor-link"></a><a href="#consumer_rebalance_protocol_upgrade">Upgrade & Downgrade</a></h4>
+
+ <h5 class="anchor-heading"><a id="consumer_rebalance_protocol_upgrade_offline" class="anchor-link"></a><a href="#consumer_rebalance_protocol_upgrade_offline">Offline</a></h5>
+
+ <p>Consumer groups are automatically converted from <code>Classic</code> to <code>Consumer</code> and vice versa when they are empty. Hence, it is possible to change the protocol
+ used by the group by shutting down all the consumers and bringing them back up with the <code>group.protocol=consumer</code> configuration. The downside is that it requires taking
+ the consumer group down.</p>
+
+ <h5 class="anchor-heading"><a id="consumer_rebalance_protocol_upgrade_online" class="anchor-link"></a><a href="#consumer_rebalance_protocol_upgrade_online">Online</a></h5>
+
+ <p>Consumer groups can be upgraded without downtime by rolling out the consumer with the <code>group.protocol=consumer</code> configuration. When the first consumer using
+ the new consumer rebalance protocol joins the group, the group is converted from <code>Classic</code> to <code>Consumer</code>, and the classic rebalance protocol is
+ interoperated to work with the new consumer rebalance protocol. This is only possible when the classic group uses an assignor that does not embed custom metadata.</p>
+
+ <p>Consumer groups can be downgraded using the opposite process. In this case, the group is converted from <code>Consumer</code> to <code>Classic</code> when the last
+ consumer using the new consumer rebalance protocol leaves the group.</p>
+
+ <h4 class="anchor-heading"><a id="consumer_rebalance_protocol_limitations" class="anchor-link"></a><a href="#consumer_rebalance_protocol_limitations">Limitations</a></h4>
+
+ <p>While the new consumer rebalance protocol works for most use cases, it is still important to be aware of the following limitations:</p>
+ <ul>
+ <li>Client-side assignors are not supported. (see <a href="https://issues.apache.org/jira/browse/KAFKA-18327">KAFKA-18327</a>)</li>
+ <li>Rack-aware assignment strategies are not fully supported. (see <a href="https://issues.apache.org/jira/browse/KAFKA-17747">KAFKA-17747</a>)</li>
+ </ul>
+
</script>
<div class="p-ops"></div>
diff --git a/docs/upgrade.html b/docs/upgrade.html
index 3c42da28214..8e4b3f02cee 100644
--- a/docs/upgrade.html
+++ b/docs/upgrade.html
@@ -45,6 +45,21 @@
latest Kafka releases.
<!--#include virtual="zk2kraft.html" -->
</li>
+ <li>
+ Apache Kafka 4.0 ships with a brand-new group coordinator implementation (See <a href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217387038#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-GroupCoordinator">here</a>.
+ Functionally speaking, it implements all the same APIs. However, the group coordinator runs in its own dedicated threads.
+ The number of threads can be defined with the <code>group.coordinator.threads</code> configuration. It defaults to 1 and
+ must certainly be adjusted for production deployments.
+ </li>
+ <li>
+ The Next Generation of the Consumer Rebalance Protocol (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol">KIP-848</a>)
+ is now Generally Available (GA) in Apache Kafka 4.0. The protocol is automatically enabled on the server when the upgrade to 4.0 is finalized.
+ Note that once the new protocol is used by consumer groups, care must be taken to downgrade to a previous version. Downgrading
+ to version 3.4.1 or above is safe. Downgrading to an earlier version requires to ensure that all the records in the
+ <code>__consumer_offsets</code> are removed because those versions cannot handle them. This can be achieved by deleting all the
+ consumer groups using the new protocol, disabling the new protocol and letting compaction remove the tombstones.
+ Check <a href="/{{version}}/documentation.html#consumer_rebalance_protocol">here</a> for details.
+ </li>
<li>A number of deprecated classes, methods, configurations and tools have been removed.
<ul>
<li><b>Common</b>
@@ -285,7 +300,7 @@
See <a href="https://issues.apache.org/jira/browse/KAFKA-17531">KAFKA-17531</a> for details.
</li>
<li>
- The most important changes are highlighted in the <a href=""/{{version}}/documentation/streams/upgrade-guide.html#streams_api_changes_400">Kafka Streams upgrade guide</a>.
+ The most important changes are highlighted in the <a href="/{{version}}/documentation/streams/upgrade-guide.html#streams_api_changes_400">Kafka Streams upgrade guide</a>.
</li>
<li>
For a full list of changes, see <a href="https://issues.apache.org/jira/browse/KAFKA-12822">KAFKA-12822</a>.